Methods and systems for codec detection in video streams

ABSTRACT

Method and apparatus for carrying out the method receiving packets, each of the packets comprising a header and a payload. For a particular packet among the packets, the method includes processing at least the header of the particular packet to determine a flow associated with the particular packet; attempting to determine a payload structure based on the flow, the payload structure associated with transport of coded video data in the payload of the particular packet; and if the attempting is successful, repackaging coded video data contained in the payload of the particular packet into a new packet and forwarding the new packet to an external system or storing the new packet in memory.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of (i) U.S. Provisional PatentApplication Ser. No. 62/850,788 filed on May 21, 2019; (ii) U.S.Provisional Patent Application Ser. No. 63/013,021 filed on Apr. 21,2020; and (iii) U.S. Provisional Patent Application Ser. No. 63/027,217filed on May 19, 2020, all of which are hereby incorporated by referenceherein.

FIELD

This disclosure relates generally to the field of digital video and,more particularly, to methods and systems for codec detection in videostreams.

BACKGROUND

Building security systems typically include a closed-circuit networkbetween a set of cameras connected to a switch, and a video managementsystem (or server) connected to the switch. The cameras can use any of avariety of encoders to encode the video images into a particular format(e.g., H263, MPEG4, H.264) for transmission to the switch in the form ofpackets.

Should the need to monitor or intercept these packets arise, e.g., forlaw enforcement purposes, personnel entering the building in aclandestine fashion may gain access to the communication link betweenthe switch and the VMS. However, there is little or no a prioriknowledge of the encoders used to encode the various video streamstraveling on the communication link. As a result, one may resort tobrute force methods, whereby multiple encoders of different types arerun in parallel and the one that produces the most coherent output isselected. However, this becomes a computationally intensive approach andis unwieldy, requiring large or heavy equipment to be brought into thebuilding.

The complexity of the problem is exacerbated as the number of camerasgrows, since this results in an increase in the bit rate of thecommunication link between the switch and the VMS. As a result, lawenforcement and other interested third parties would welcome an approachthat allows more rapid and computationally efficient detection of videostreams.

SUMMARY

According to a first broad aspect, there is provided acomputer-implemented method, comprising:

-   -   receiving packets, each of the packets comprising a header and a        payload;    -   for each particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   processing at least part of the payload of the particular            packet to determine a candidate payload structure of the            particular packet and processing at least part of the            payload of the particular packet in accordance with the            candidate payload structure, which includes processing at            least part of the payload of the particular packet in            accordance with one or more codec-specific tests; and        -   in case a given test of the one or more codec-specific tests            is passed, creating an association between the flow            associated with the particular packet and the candidate            payload structure.

According to another broad aspect, there is provided a computing devicecomprising:

-   -   a computer-readable program storage unit comprising an        application program and an operating system code; and    -   a processor being configured to read and execute the application        program so as to carry out a method that comprises:        -   receiving packets, each of the packets comprising a header            and a payload; and        -   for each particular packet among the packets:            -   processing at least the header of the particular packet                to determine a flow associated with the particular                packet;            -   processing at least part of the payload of the                particular packet to determine a candidate payload                structure of the particular packet and to process at                least part of the payload of the particular packet in                accordance with the candidate payload structure, which                includes processing at least part of the payload of the                particular packet in accordance with one or more                codec-specific tests;            -   creating an association between the flow associated with                the particular packet and the candidate payload                structure if a given test of the one or more                codec-specific tests is passed.

According to another broad aspect, there is provided a computer-readablemedium storing computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:

-   -   receiving packets, each of the packets comprising a header and a        payload; and    -   for each particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   processing at least part of the payload of the particular            packet to determine a candidate payload structure of the            particular packet and processing at least part of the            payload of the particular packet in accordance with the            candidate payload structure, which includes processing at            least part of the payload of the particular packet in            accordance with one or more codec-specific tests; and        -   in case a given test of the one or more codec-specific tests            is passed, creating an association between the flow            associated with the particular packet and the candidate            payload structure.

According to another broad aspect, there is provided acomputer-implemented method, comprising:

-   -   receiving a packet, the packet comprising a header and a        payload;    -   processing a portion of the packet to identify a flow associated        with the packet; and    -   determining whether the payload contains pre-determined        codec-identifying information regarding a particular codec that        is sufficient to identify the particular codec as having been        used to encode video data in the payload and, if so, returning        the particular codec as an identified codec for the flow.

According to another broad aspect, there is provided a computing devicecomprising:

-   -   a computer-readable program storage unit comprising an        application program and an operating system code; and    -   a processor being configured to read and execute the application        program so as to carry out a method that comprises:        -   receiving a packet, the packet comprising a header and a            payload;        -   processing a portion of the packet to identify a flow            associated with the packet; and        -   determining whether the payload contains pre-determined            codec-identifying information regarding a particular codec            that is sufficient to identify the particular codec as            having been used to encode video data in the payload and, if            so, to return the particular codec as an identified codec            for the flow.

According to another broad aspect, there is provided a computer-readablemedium storing computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:

-   -   receiving a packet, the packet comprising a header and a        payload;    -   processing a portion of the packet to identify a flow associated        with the packet; and    -   determining whether the payload contains pre-determined        codec-identifying information regarding a particular codec that        is sufficient to identify the particular codec as having been        used to encode video data in the payload and, if so, returning        the particular codec as an identified codec for the flow.

According to another broad aspect, there is provided acomputer-implemented method, comprising:

-   -   receiving a packet, the packet comprising a header and a        payload;    -   processing a portion of the packet to identify a flow associated        with the packet; and    -   determining whether the payload contains partial        codec-identifying information regarding a particular codec and,        if so:    -   adding the partial codec-identifying information to previously        determined partial codec-identifying information regarding the        particular codec to obtain cumulative codec-identifying        information regarding the particular codec; and    -   returning the particular codec as an identified codec for the        flow in case the cumulative codec-identifying information        regarding the particular codec is sufficient to identify the        particular codec as having been used to encode video data in the        payload.

According to another broad aspect, there is provided a computing devicecomprising:

-   -   a computer-readable program storage unit comprising an        application program and an operating system code; and    -   a processor being configured to read and execute the application        program so as to carry out a method that comprises:        -   receiving a packet, the packet comprising a header and a            payload;        -   processing a portion of the packet to identify a flow            associated with the packet; and        -   determining whether the payload contains partial            codec-identifying information regarding a particular codec            and, if so:        -   adding the partial codec-identifying information to            previously determined partial codec-identifying information            regarding the particular codec to obtain cumulative            codec-identifying information regarding the particular            codec; and        -   returning the particular codec as an identified codec for            the flow in case the cumulative codec-identifying            information regarding the particular codec is sufficient to            identify the particular codec as having been used to encode            video data in the payload.

According to another broad aspect, there is provided a computing devicecomprising a computer-readable medium storing computer-readableinstructions which, when executed by a processor, cause the processor tocarry out a method that comprises:

-   -   receiving a packet, the packet comprising a header and a        payload;    -   processing a portion of the packet to identify a flow associated        with the packet; and    -   determining whether the payload contains partial        codec-identifying information regarding a particular codec and,        if so:    -   adding the partial codec-identifying information to previously        determined partial codec-identifying information regarding the        particular codec to obtain cumulative codec-identifying        information regarding the particular codec; and    -   returning the particular codec as an identified codec for the        flow in case the cumulative codec-identifying information        regarding the particular codec is sufficient to identify the        particular codec as having been used to encode video data in the        payload.

According to another broad aspect, there is provided acomputer-implemented method, comprising:

-   -   receiving packets, each of the packets comprising a header and a        payload;    -   for a particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   attempting to determine a payload structure based on the            flow, the payload structure associated with transport of            coded video data in the payload of the particular packet;        -   if the attempting is successful, repackaging coded video            data contained in the payload of the particular packet into            a new packet and forwarding the new packet to an external            system or storing the new packet in memory.

According to another broad aspect, there is provided a computing devicecomprising:

-   -   a computer-readable program storage unit comprising an        application program and an operating system code; and    -   a processor being configured to read and execute the application        program so as to carry out a method that comprises:    -   receiving packets, each of the packets comprising a header and a        payload;    -   for a particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   attempting to determine a payload structure based on the            flow, the payload structure associated with transport of            coded video data in the payload of the particular packet;        -   if the attempting is successful, repackaging coded video            data contained in the payload of the particular packet into            a new packet and forwarding the new packet to an external            system or storing the new packet in memory.

According to another broad aspect, there is provided a computer-readablemedium storing computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:

-   -   receiving packets, each of the packets comprising a header and a        payload;    -   for a particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   attempting to determine a payload structure based on the            flow, the payload structure associated with transport of            coded video data in the payload of the particular packet;        -   if the attempting is successful, repackaging coded video            data contained in the payload of the particular packet into            a new packet and forwarding the new packet to an external            system or storing the new packet in memory.

According to another broad aspect, there is provided a systemcomprising:

-   -   a tap for capturing packets sent from a source device to a        destination device on a data network;    -   a surveillance module operatively coupled to the tap, the        surveillance module having an input port, and output port and a        processing entity configured for receiving the captured packets        from the tap via the input port, each of the packets comprising        a header and a payload and, for a particular packet among the        packets:        -   (i) processing at least the header of the particular packet            to determine a flow associated with the particular packet;        -   (ii) attempting to determine a payload structure based on            the flow, the payload structure associated with transport of            coded video data in the payload of the particular packet;        -   (iii) if the attempting is successful, repackaging coded            video data contained in the payload of the particular packet            into a new packet and forwarding the new packet to an            external system via the output port or storing the new            packet in memory.

According to another broad aspect, there is provided acomputer-implemented method, comprising:

-   -   intercepting packets sent from a source device and destined for        a destination device, each of the packets comprising a header        and a payload;    -   for a particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   attempting to determine a payload structure and a codec            based on the flow;        -   if the attempting is successful, replacing at least part of            the payload of the particular packet with replacement coded            video data that has been encoded with the codec, the packet            with the replacement coded video data being then released            towards the destination device instead of the particular            packet.

According to another broad aspect, there is provided a computing devicecomprising:

-   -   a computer-readable program storage unit comprising an        application program and an operating system code; and    -   a processor being configured to read and execute the application        program so as to carry out a method that comprises:    -   intercepting packets sent from a source device and destined for        a destination device, each of the packets comprising a header        and a payload;    -   for a particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   attempting to determine a payload structure and a codec            based on the flow;        -   if the attempting is successful, replacing at least part of            the payload of the particular packet with replacement coded            video data that has been encoded with the codec, the packet            with the replacement coded video data being then released            towards the destination device instead of the particular            packet.

According to another broad aspect, there is provided a computer-readablemedium storing computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:

-   -   intercepting packets sent from a source device and destined for        a destination device, each of the packets comprising a header        and a payload;    -   for a particular packet among the packets:        -   processing at least the header of the particular packet to            determine a flow associated with the particular packet;        -   attempting to determine a payload structure and a codec            based on the flow;        -   if the attempting is successful, replacing at least part of            the payload of the particular packet with replacement coded            video data that has been encoded with the codec, the packet            with the replacement coded video data being then released            towards the destination device instead of the particular            packet.

According to another broad aspect, there is provided aprocessor-implemented method, comprising:

-   -   receiving a plurality of packets, each belonging to one of a        plurality of flows, each packet comprising a header and a        payload;    -   responsive to receipt of each of the packets:        -   searching in a memory for a record associated with the flow            to which the packet belongs;        -   in case the searching finds no record in the memory,            allocating a portion of the memory to a record associated            with the flow to which the packet belongs;        -   attempting codec identification by processing at least part            of the payload of the packet;        -   in case the attempting successfully identifies a particular            codec, storing information regarding the particular codec in            the record associated with the flow to which the packet            belongs;        -   in case the attempting is unsuccessful and a certain            condition has been reached since the portion of the memory            has been allocated, freeing up the portion of the memory.

According to another broad aspect, there is provided a computer-readablemedium storing computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:

-   -   receiving a plurality of packets, each belonging to one of a        plurality of flows, each packet comprising a header and a        payload;    -   responsive to receipt of each of the packets:        -   searching in a memory for a record associated with the flow            to which the packet belongs;        -   in case the searching finds no record in the memory,            allocating a portion of the memory to a record associated            with the flow to which the packet belongs;        -   attempting codec identification by processing at least part            of the payload of the packet;        -   in case the attempting successfully identifies a particular            codec, storing information regarding the particular codec in            the record associated with the flow to which the packet            belongs;        -   in case the attempting is unsuccessful and a certain            condition has been reached since the portion of the memory            has been allocated, freeing up the portion of the memory.

According to another broad aspect, there is provided aprocessor-implemented method, comprising:

-   -   receiving a data stream containing video data for a given flow;    -   receiving a control stream that contains codec-identifying        information associated with the given flow; and    -   creating an output stream including packets containing the video        data from the data stream and additional packets containing the        codec-identifying information.

According to another broad aspect, there is provided a computer-readablemedium storing computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:

-   -   receiving a data stream containing video data for a given flow;    -   receiving a control stream that contains codec-identifying        information associated with the given flow; and    -   creating an output stream including packets containing the video        data from the data stream and additional packets containing the        codec-identifying information.

According to another broad aspect, there is provided a method carriedout by a device for connection to a network that supports communicationof packets between at least one first entity and at least one secondentity, the method comprising:

-   -   receiving the packets;    -   identifying, based on information contained in at least a        payload of respective ones of the received packets, those of the        received packets that are packets of interest;    -   grouping the payloads of the packets of interest into streams of        packets, the payloads of those of the received packets that are        not packets of interest not being so grouped; and    -   transmitting the streams of packets to a third destination for        processing, the third destination being different from the first        and second entities.

According to another broad aspect, there is provided a method thatcomprises:

-   -   identifying, in a plurality of received packets conveying        frames, certain ones of the frames that are related to one        another;    -   identifying, in the related frames, portions of the related        frames that are in accordance with at least one structure of        interest;    -   assembling said portions of the related frames into at least one        stream of new packets; and    -   storing the at least one stream of new packets in memory or        sending the at least one stream of new packets to an external        entity.

According to another broad aspect, there is provided a non-transitorycomputer-readable storage medium storing computer-readable instructionswhich, when read and executed by a processor of a computing entity,cause the computing entity to carry out a method that comprises:

-   -   identifying, in a plurality of received packets conveying        frames, certain ones of the frames that are related to one        another;    -   identifying, in the related frames, portions of the related        frames that are in accordance with at least one structure of        interest;    -   assembling said portions of the related frames into at least one        stream of new packets; and    -   storing the at least one stream of new packets in memory or        sending the at least one stream of new packets to an external        entity.

BRIEF DESCRIPTION OF THE DRAWINGS

A detailed description of embodiments is provided below, by way ofexample only, with reference to drawings accompanying this description,in which:

FIGS. 1A, 1B and 10 are block diagrams showing examples of a videonetwork architecture.

FIG. 2 is a block diagram of an example surveillance module.

FIG. 3 shows an IP packet comprising a header and a payload.

FIG. 4A shows an example of a UDP packet carrying coded video data.

FIG. 4B shows an example of a TCP packet carrying coded video data.

FIGS. 5 and 5A are flowcharts corresponding to two variants of anexample codec detection process.

FIG. 6 is a flowchart corresponding to an example payload redirectionprocess.

FIG. 7 is a conceptual diagram of the example payload redirectionprocess.

FIG. 8A is a flowchart corresponding to an example codec autodetectsub-process with an input variable “UDP”.

FIG. 8B to 8F are flowcharts corresponding to various example testsdesigned to confirm whether an RTP payload is coded with a particularcodec.

FIGS. 8G and 8H are parts of a flowchart corresponding to an examplecodec autodetect sub-process with the input variable “TOP”.

FIG. 9A is a block diagram of example memory container.

FIG. 9B is a flowchart corresponding to an example memory managementprocess.

FIG. 10 is block diagram of an example surveillance module.

FIG. 11 is a flowchart corresponding to an example payload modificationprocess.

FIG. 12 is a block diagram of an example computing device.

FIG. 13 is a flowchart corresponding to an example method carried outwithin the codec detection module.

FIGS. 14 and 15 are flowcharts corresponding to example codec autodetectsub-processes.

FIG. 16 is a flowchart corresponding to an example method carried outwithin a payload redirection module.

FIG. 17 is a flowchart corresponding to an example method carried outwithin a payload modification module.

FIG. 18 is a flowchart corresponding to an example memory managementprocess.

FIGS. 19 and 20 illustrate in detail an out-of-band codec controlinformation reconstruction process, in accordance with a non-limitingembodiment.

FIG. 21 is a flowchart illustrating an example codec control informationreconstruction process.

FIG. 22 illustrates in detail an out-of-band codec control informationreconstruction process, in accordance with another non-limitingembodiment.

FIG. 23A is a block diagram showing a surveillance module that obtainsreceived packets.

FIG. 23B conceptually illustrates related frames carried by the receivedpackets.

FIGS. 24A to 24F show different structures of interest that can becarried by related frames.

FIG. 25 is a tree showing structures of interest.

FIG. 26 is a table showing constraints that lead to throttling of theamount of collected data that is recorded in memory or data sent to anexternal entity.

It is to be expressly understood that the description and drawings areonly for purposes of illustrating certain embodiments and are an aid forunderstanding. They are not intended to be and should not be limiting.

DETAILED DESCRIPTION

FIGS. 1A and 1B show a video network architecture 10 in which one ormore cameras 12 ₁ . . . 12 _(A) are connected by a network 14 to aswitch 16. The network 14 may be a closed-circuit private video network,such as may be provided in a building (e.g., condominium tower, officetower, warehouse, airport terminal, school, etc.). The cameras 12 ₁ . .. 12 _(A) capture images, which are encoded into blocks of coded videodata 22 in accordance with a particular codec format such as H263,MPEG4, H.264, H.264 bitstream mode, H.265, AAC, PCM and MJPEG, to name afew non-limiting possibilities. In a non-limiting embodiment, thecameras 12 ₁ . . . 12 _(A) may be Internet Protocol (IP) cameras, whichproduce IP packets that carry the blocks of coded video data 22. Blocksof coded video data 22 representing sequential images originating fromthe same camera may be referred to as a video stream. The rate at whicha camera produces new images may vary, from e.g., 1 frame every fewseconds to e.g., 24 frames per second or more. This, together with thequality of the image and the codec used, has an impact on the bit rateof each video stream.

The switch 16 is connected to a video management system (VMS) 26 by acommunication link 28. The communication link 28 may thus carry numerousvideo streams from the cameras 12 ₁ . . . 12 _(A), and such videostreams may be interleaved or multiplexed in various ways on thecommunication link 28. The communication link 28 may be an Ethernetlink, such as 100 MB Ethernet (e.g., Category 5), 10 GB Ethernet (e.g.,Category 7), etc. In such an embodiment, the switch exchanges Ethernetframes with the VMS 26. Some of the Ethernet frames traveling in thedirection from the switch 16 to the VMS 26 encapsulate packets whichcontain coded video data 22 representing images captured by the cameras,whereas other such Ethernet frames may include control information(e.g., status information relating to the cameras or the switch). In theopposite direction, some of the Ethernet frames traveling from the VMS26 to the switch 16 may encapsulate packets which contain controlinformation (e.g., information for controlling the cameras or theswitch). If the switch 16 and the VMS 26 are members of and connected toa local area network (now shown), additional devices may be connected tothis local area network, which could mean that additional packets travelbetween the switch 16 and the VMS 26 in one direction or the other. Suchadditional packets may include additional data that might not be codedvideo data 22 from any of the cameras 12 ₁ . . . 12 _(A).

The packets encapsulated in the Ethernet frames traveling on thecommunication link 28 may be Internet Protocol (IP) packets 20 (e.g.,IPv4 or IPv6 packets). With reference to FIG. 3, an IP packet 20 isshown to comprise a header 34 and a payload 36. The header 34 specifies,inter alia, a source IP address and a destination IP address. The header34 may also include a medium access control (MAC) address.

A (software) application at the cameras 12 ₁ . . . 12 _(A) (or at theswitch 16) determines what information goes into the headers of the IPpackets so as to allow proper delivery of the packet to a destinationacross a network 14. Also, the (software) application at the cameras 12₁ . . . 12 _(A) (or at the switch 16) determines the communicationsprotocols to be used for transmission of the coded video data 22 withinthe payloads 36 of the IP packets 20. Such communications protocols mayinclude a lower-layer communications protocol and a higher-layercommunications protocol.

The lower-layer communications protocol may be connectionless or may beconnection-oriented. One non-limiting example of a connectionlesslower-layer communications protocol is the User Datagram Protocol (UDP).One non-limiting example of a connection-oriented lower-layercommunications protocol is the Transmission Control Protocol (TCP). Assuch, the payload 36 of an IP packet 20 may include one or more UDPpackets or TCP packets.

Each UDP or TCP packet has its own packet format, including a header anda payload, as will now be described.

In the case of a UDP packet 48, and with reference to FIG. 4A, theheader includes control information relating to the UDP packet 48, suchas a source port and a destination port. For its part, the payload of aUDP packet 48 includes data, which may be of various types.

In one example, the data carried by the payload of the UDP packet 48 isvideo information. In that case, the payload of a UDP packet 48 mayinclude blocks of coded video data 22 that are formatted in accordancewith a higher-layer communications protocol. An example of such ahigher-layer communications protocol is RTP. Thus, for example, thepayload of a UDP packet 48 may include an RTP packet with an RTP header54 and an RTP payload 56. The RTP header 54 includes control informationrelating to the RTP packet, whereas the RTP payload 56 includes blocksof coded video data 22.

Therefore, in summary, coded video data 22 pertaining to a particularvideo stream occupies the RTP payload of an RTP packet 48 which,together with the RTP header, is encapsulated within the payload of aUDP packet 48 which, together with its own header, is encapsulatedwithin the payload 36 of an IP packet 20. This can be referred to as anRTP-over-UDP (or RTP/UDP) payload structure 60.

Turning now to the case of a TCP packet 62, and with reference to FIG.4B, the header includes control information relating to the TCP packet62, such as a source port 64 and a destination port 66. For its part,the payload of a TCP packet 62 includes data, which may be of varioustypes.

In one example, the data carried by the payload of the TCP packet 62 isvideo information. In that case, the payload of a TCP packet 62 mayinclude coded video data 22 that is formatted in accordance with ahigher-layer communications protocol. Examples of such a higher-layercommunications protocol include Hypertext Transport Protocol (HTTP) andRTSP-Interleaved (RTSP-I).

Thus, for example, in the case of HTTP, the payload of the TCP packet 62may include an introductory portion (e.g., --*\r\nContent-Type:image/jpeg\r\nContent-Length*\r\n\r\n) followed by blocks of coded videodata 22 in JPEG format. This is referred to as an “HTTP MJPEG multipart”payload structure 68.

In the case of RTSP-I, the payload of the TCP 62 packet may include asequence of RTP headers 54 and corresponding RTP payloads 56. The RTPheaders 54 are associated with respective video streams and includecontrol information relating to the corresponding RTP payloads 56, andthe RTP payload 56 includes blocks of coded video data 22 for therespective video streams. Therefore, coded video data 22 pertaining tomultiple video streams occupies the payloads of multiple RTP packetswhich, together with their corresponding headers, are encapsulatedwithin the payload of a TCP packet 62 which, together with its ownheader, is encapsulated within the payload 36 of an IP packet 20. Thisis referred to as an RTSP-I-over-TCP (or RTSP-I/TCP) payload structure69.

In another example, the data carried by the payload of the TCP packet 62is control information. The control information may be used for variouspurposes, such as, for example, to set up or control the transmission ofvideo information. For instance, the payload of the TCP packet mayinclude control information, which itself may be arranged to follow aparticular control protocol. An example of such a control protocol isthe “Real Time Streaming Protocol” (RTSP)—see RFC 2326 or 7826, herebyincorporated by reference herein. Control information sent in accordancewith RTSP may include commands such as DESCRIBE, SETUP, TERADOWN, PLAYand PAUSE, for example.

It is noted that the IP packets 20 produced by different ones of thecameras 12 ₁ . . . 12 _(A) may have different payload structures,depending on the manufacturer, model or camera setting. As such, the IPpackets 20 produced by some of the cameras 12 ₁ . . . 12 _(A) may havean RTP/UDP payload structure 60, whereas the IP packets 20 produced byother ones of the cameras 12 ₁ . . . 12 _(A) may have an RTSP-I/TCPpayload structure 69, and the IP packets 20 produced by still other onesof the cameras 12 ₁ . . . 12 _(A) may have the HTTP MJPEG multipartpayload structure 68. Still other payload structures may be used tocarry coded video data 22 from the cameras to the VMS 26 via the switch16.

For the purposes of this disclosure, each video stream generated by oneof the cameras 12 ₁ . . . 12 _(A) may be characterized by a unique“flow”. The flow may be defined by any suitable combination ofidentifiers. Such identifiers could be found exclusively in the IPheader or they could be found partly in the IP header and partly in theIP payload, such as in the header of a TCP or UDP packet carried by theIP payload. For example, in the case of an IP packet carrying a TCPpacket in the IP payload, the flow may be defined by identifiers in theIP header and the TCP header. In the case of an IP packet carrying a UDPpacket in the IP payload, the flow may be defined by identifiers in theIP header and the UDP header. Thus, the flow associated with an IPpacket 20 may be defined by header information that appears in both theIP header of the IP packet 20 and in the header of whichever sub-packetis encapsulated within the IP payload of the IP packet 20. In someembodiments, the flow can be a unique combination of header informationchosen from source address, destination address, source port,destination port and MAC address (to name a few possible identifiers).In some cases, a flow uniquely identifies a single video stream from asingle camera.

In accordance with the present disclosure, and with reference to FIGS.1A, 1B, 2 and 10, a surveillance module 70 is provided. In general,there are two main embodiments of the surveillance module 70, one forlistening (see FIGS. 1A, 1B and 2) and one for modification (see FIG.10). Each of these main embodiments will be described in turn.

Case 1—Listening (“Passive”)

In an embodiment, a passive tap 74 may be connected to the communicationlink 28. A passive tap 74 does not interrupt or inspect the passage ofdata along the communication link 28. FIGS. 1A and 1B show twonon-limiting embodiments of a passive tap 74 Specifically, FIG. 1A showsthe example of a tap 74 in the case where the communication link 28between the switch 16 and the VMS 26 includes two unidirectional linksand FIG. 1B shows the example of a tap 74 in the case where thecommunication link 28 between the switch 16 and the VMS 26 includes abidirectional link.

In the case of unidirectional sub-links (namely, a switch-to-VMSsub-link and a VMS-to-switch sub-link), and with reference to FIG. 1A,the tap 74 comprises a connector 152 that is electrically coupled to theswitch-to-VMS sub-link and an output port leading to the surveillancemodule 70. In order to electrically couple the connector 152 to theswitch-to-VMS sub-link, non-contact techniques may be used (e.g., anopto-isolator). Alternatively, a portion of the insulation of theswitch-to-VMS sub-link may be stripped, exposing a wire, and theconnector 152 may be attached to the exposed wire. The connector 152allows the transfer of video-data-containing packets from the cameras 12₁ . . . 12 _(A) to the surveillance module 70. Still other ways ofmirroring the in-transit data signal may be used. Optionally, a secondconnector 152 may be provided that is electrically coupled to theVMS-to-switch sub-link and also leading directly to the surveillancemodule 70. This second connector 152 would allow the surveillance module70 to receive and process packets sent by the VMS 26. Wirelessinterception techniques may also be used in some embodiments.

In the case of a bidirectional link, and with reference to FIG. 1B, thetap 74 comprises a switch-side hybrid 100 with an input/output port130S, an input port 140S and an output port 150S, and a VMS-side hybrid110 with an input/output port 130V, an input port 140V and an outputport 150V. The VMS-side hybrid 110 separates signals on the input/outputport 130V that are simultaneously traveling to and from the VMS 26. Assuch, signals coming from the VMS 26 and arriving at the input/outputport 130V of the VMS-side hybrid 110 (e.g., via an RJ45 connector) aresent to the output port 150V of the VMS-side hybrid 110V, whereas theinput/output port 130V of the VMS-side hybrid 110 also carries signalsarriving at the VMS-side hybrid 110 via its input port 140V. Similarly,the switch-side hybrid 100 separates signals on the input/output port130S of the switch-side hybrid 100 that are simultaneously traveling toand from the switch 16. As such, signals coming from the switch 16 andarriving at the input/output port 130S of the switch-side hybrid 100(e.g., via an RJ45 connector) are sent to the output port 150S of theswitch-side hybrid 100, whereas the input/output port 130S of theswitch-side hybrid 100 also carries signals arriving at the switch-sidehybrid 100 via the input port 140S of the switch-side hybrid 100.

In this embodiment, the (passive) tap 74 comprises a connector 152coupled to the connection between the output port 150S of theswitch-side hybrid 100 and the input port 140V of the VMS-side hybrid110. The connector 152 may have an output port leading to thesurveillance module 70. Optionally, a second connector 152 may beprovided that is electrically coupled between the output port 150V ofthe VMS-side hybrid 110 and the input port 140S of the switch-sidehybrid 100. This second connector 152 would allow the surveillancemodule 70 to receive and process packets sent by the VMS 26. Wirelessinterception techniques may be used, depending on operationalrequirements. In order to connect the tap 74, the communication link 28may be disconnected or broken, and the two resultant ends are fed to theinput/output ports 130S, 130V of the two hybrids. In some operationalcontexts, severing of the communication link 28 may be donesurreptitiously, e.g., by law enforcement personnel unbeknownst to theowner or manager of the communication link 28 between the switch 16 andthe VMS 26.

It is also within the scope of the present disclosure to use an activetap, i.e., one that interrupts and inspects the passage of data on thecommunication link 28. In fact, for some types of communication links(e.g., Gigabit Ethernet), an active tap may be a preferred way to accessthe data on the communication link 28, due to the use of differentialvoltages for transmitting the data, which may make passive tappingdifficult. Embodiments for the active tap include Ethernet PHY tapping,port mirroring and use of a managed switch.

In the case of an active tap, and with reference to FIG. 10, aman-in-the-middle module (MMM) 99 is configured to receive signals fromthe output port of the switch-side hybrid 100, extract Ethernet frames(that encapsulate IP packets), write the packets to an internal memory,read the packets from the internal memory 95 and send them towards both(i) the input port of the VMS-side hybrid 110 and (ii) an output port 97leading to the surveillance module 70. The MMM 99 thus allows copies ofthe same video-data-containing packets from the cameras 12 ₁ . . . 12_(A) to be fed to the VMS 26 and to the surveillance module 70.

In addition, the MMM 99 may be configured to similarly intercept andreplicate Ethernet frames received from the VMS 26 and destined for thecameras 12 ₁ . . . 12 _(A). In such a case, the MMM 99 may allow thesurveillance module 70 to receive and monitor packets (e.g., includingcontrol information) sent by the VMS 26.

Also, the MMM 99 may be configured to insert additional packets destinedfor the cameras 12 ₁ . . . 12 _(A). Such additional packets may carrycontrol messages to the cameras 12 ₁ . . . 12 _(A) in order to cause thecameras 12 ₁ . . . 12 _(A) to take certain actions. For example, thecontrol messages may include an RTSP DESCRIBE message which, uponreceipt by a camera, may cause the camera to negotiate transmission of avideo stream. This negotiation may contain crucial control informationthat may be detected and used by the surveillance module 70.

Turning now to FIG. 2, the surveillance module 70 may be operativelycoupled to an external system 72. In some embodiments, the externalsystem 72 may be implemented as a console as shown in FIGS. 1A and 1B.In other embodiments, the external system 72 may be implemented as adisplay 192 while in still other embodiments, the external system 72 maybe implemented as a data network 194 (such as the Internet) or a serverwith an address on such data network 194. Still other implementations ofthe external system 72 are possible. The external system 72 may becommunicatively isolated from the private network between the cameras 12₁ . . . 12 _(A) and the switch 16, as well as from the communicationlink 28 between the switch 16 and the VMS 26.

Those skilled in the art will appreciate that the incoming packets(e.g., IP packets 20) at the surveillance module 70 may arrive at a highspeed (packet bit rate), and in some cases may include coded video data22 from one or more of the cameras 12 ₁ . . . 12 _(A), whereas in somecases the incoming packets may include data that is not coded video data22 (e.g., control information related to the cameras 12 ₁ . . . 12 _(A)or data not related in any way to the cameras 12 ₁ . . . 12 _(A)). Therole of the surveillance module 70 is to identify, among theunpredictable morass of received IP packets 20, those containing codedvideo data 22 and to send the coded video data 22 onwards to theexternal system 72 for storage, decoding and/or analysis (or store themin a memory (not shown)). This is not a simple task, as the surveillancemodule 70 does not know a priori whether any given received packet is avideo-data-containing packet. As such, the surveillance module 70 isconfigured to process received IP packets 20 on-the-fly with a view toidentify those that contain coded video data 22, to group them on aper-flow basis and to send them to the external system 72 (or store themin a memory).

To this end, the surveillance module 70 is configured for receiving viaan input port a signal supplied by the tap 74. The received signal isfed to a packet capture module 76. The packet capture module 76 isconfigured to detect packets (e.g., IP packets 20) in the receivedsignal. The packets detected by the packet capture module 76 arefed/copied to both (i) a codec detection module 78 and (ii) a payloadredirection module 82.

The codec detection module 78 carries out a codec detection process 222which is now described generally with reference to the flowchart in FIG.5. At step 1010, an IP packet is received from the packet capture module76. A purpose of the codec detection process 222 may be to discriminatebetween two main possibilities: (1) the packet contains coded video dataand (2) the packet does not contain coded video data. If the packetcontains coded video data, then this means that the packet would have acertain payload structure, and therefore the outcome of the codecdetection process 222 should be the flow of the packet as well as thecorresponding payload structure (if correctly determined), which shouldthen allow the payload redirection module 82 to properly deconstruct andredirect the packets it receives for that flow. On the other hand, ifthe packet does not contain coded video data (or contains coded videodata that the codec detection module 78 is unable to detect), then theoutcome of the codec detection process 222 would be either nothing or anindication of the flow of the packet together with an indication thatthere was no detectable coded video in the packet.

Accordingly, at step 1020, the codec detection process 222 includesdetermining whether the received packet is an IP packet. If not, thepacket may be ignored or discarded. If yes, the next step is step 1030,whereby the codec detection process 222 includes attempting to check thelower-layer communication protocol used by the IP packet. If none can beidentified, then the process may terminate. In case the communicationlower-layer communication protocol is UDP, the codec detection process222 proceeds to step 1040, whereby a codec autodetect sub-process 210 iscarried out for the current flow with the input variable “UDP”; in casethe lower-layer communication protocol is TCP, the codec detectionprocess 222 proceeds to step 1050, whereby the codec autodetectsub-process 210 is carried out for the current flow with the inputvariable “TCP”.

The codec autodetect sub-process 210 will be described in greater detaillater on. For now, those skilled in the art should appreciate that thecodec autodetect sub-process 210 may include some initial processing ofthe payload of the received packet to determine a candidate payloadstructure of the received packet. This is still only a candidate payloadstructure because it is based on some initial information in the payloadof the received IP packet, which will need to be confirmed by individualcodec testing. Accordingly, this is followed by some processing of thepayload of the received packet in accordance with this candidate payloadstructure, which includes processing the payload of the received packetin accordance with one or more tests, each test associated with aspecific codec. If a given test of the one or more tests is passed, thiscan be viewed as confirming the candidate payload structure, and anassociation is created between the current flow and at least thecandidate payload structure. The output of the codec autodetectsub-process 210 (carried out at step 1040 or step 1050) is thus identityof the candidate payload structure as well as the codec for which thetest was ultimately passed (if any). In the present embodiment, thecodec detection process 222 then proceeds to step 1060, whereby itcreates an association between the current flow and this candidatepayload structure.

The association between the current flow and the candidate payloadstructure may be sent directly to the payload redirection module 82.Alternatively or in addition, a table 84 that stores this informationmay be populated by the codec detection module 78 and made accessible tothe payload redirection module 82. Specifically, the table 84 mayinclude records each containing a “flow” field that specifies theparameters of a given flow (e.g., any suitable combination of sourceaddress, destination address, source port, destination port, MACaddress, etc.), as well as a “payload structure” field that specifiesthe payload structure (e.g., RTP/UDP, RTSP-I/TCP, HTTP MJPEG multipart)that was found to be associated with the given flow.

It should be appreciated that the payload structure identified for agiven flow may be a best guess effort done by the codec detection module78 based on various bit patterns and state buildup (as will be describedlater on); as such, the payload structure associated with a given flowand stored in the table 84 may not always be accurate.

The information in the table 84 provides the payload redirection module82 with the information it needs to decide what to do with the packetsit receives from the packet capture module 76, i.e., whether to ignoreor repackage each packet. In particular, by determining the flow of aparticular received packet and consulting the table 84 for that flow,the payload redirection module determines if the “payload structure”field associated with that flow is populated in order to find out thepayload structure that is thought to be associated with the flow of theparticular received packet.

Accordingly, operation of the payload redirection module 82 may beviewed as carrying out a payload redirection process, which is nowdescribed in greater detail with reference to the flowchart in FIG. 6and the conceptual diagram in FIG. 7. In particular, at step 1070, apacket is received (it is assumed to be an IP packet, otherwise it maybe ignored or discarded, for example) and the flow is determined (basedon at least the header of the received packet but also possibly based onpart of the payload, which may contain the header of a thereinencapsulated packet). In subsequent steps (to be described below, andwhich need not be practiced in the order shown), the payload redirectionprocess behaves in a manner that depends on the payload structureassociated with the current flow, if one exists in the table 84.

For example, if the table 84 is indicative of the payload structure forthe current flow being RTP/UDP (which would mean that the received IPpacket includes a UDP packet in its payload, whereby the UDP packetincludes an RTP packet in its own payload, and whereby the RTP packetincludes an RTP header and an RTP payload), then the payload redirectionprocess includes step 1090, whereby the RTP payload is extracted andsent to a socket (such as a Berkeley socket or BSD socket or otherInternet socket or other library of linkable modules or applicationprogramming interfaces) at step 1100, where the RTP payload is packagedinto one or more suitable new packets and sent to the console or otherexternal system 72, or stored in memory for future use.

If the table 84 is indicative of the payload structure for the currentflow being RTSP-I/TCP (which would mean that the received IP packetincludes a TCP packet in its payload, whereby the TCP packet includes aninterleaved set of RTP headers and RTP payloads in its own TCP payload),then the payload redirection process includes step 1120, whereby the RTPpayloads for each flow are extracted and sent to the output socket. Atstep 1100, the socket packages the RTP payloads for each flow into newpackets which are sent to the console or other external system 72, orstored in memory for future use.

If the table 84 is indicative of the payload structure for the currentflow being HTTP MJPEG multipart (which would mean that the received IPpacket includes a TCP packet in its payload and the payload of the TCPpacket includes MJPEG coded video data in the HTTP format), then thepayload redirection process includes step 1140, whereby the payloadredirection process extracts the JPEG image(s) from the TCP payload,forms one or more RTP payloads and sends the RTP payload(s) to theoutput socket, linked library or application programming interface. Atstep 1100, the RTP payload(s) is/are packaged by the socket/library/APIinto one or more suitable new packets and sent to the console or otherexternal system 72, or stored in memory for future use.

In the aforementioned steps 1090, 1120, 1140, data that is presumed tobe coded video data is extracted from the received IP packet andrepackaged into RTP payloads sent to the output socket. This extractioncan be done “blindly”, i.e., without verifying the data or decoding it,because it is known (or believed) to obey a certain known payloadstructure.

However, in some embodiments, there may be an advantage to decoding someor all of the coded video data before repackaging it. For example, thecoded video data may include control information that should bevalidated or supplemented. For instance, the coded video data mayinclude information that signals the beginning of a video frame. It isuseful to know this from the point of view of the payload redirectionmodule 82 because the transmission of an incomplete video frame maycause artifacts. As such, it is conceivable that the payload redirectionprocess will include a step of ensuring that it does not transmit codedvideo data until it has received data signaling the beginning of a videoframe. In another example, the coded video data may includecodec-identifying information required for proper decoding (e.g., SPS,VPS). As such, it is conceivable that the payload redirection processwill include a step of ensuring that it sends the codec-identifyinginformation to be repackaged before it sends the remainder of the codedvideo data. Any missing

It should be appreciated that some of the aforementioned steps may beperformed in a different order to achieve substantially similar effects.Additional verifications of other payload structures can also be carriedout, and in any order, including in parallel.

If no payload structure was identified for the current flow, the packetmay be ignored or discarded.

It should also be noted that the new packets include the same codedvideo data as the received IP packets; the coded video data need not bedecoded, in other words new packets are sent to the console, externalsystem 72 or memory without decoding the video data. The new packets mayalso be IP packets, although this is not a requirement. If the newpackets are IP packets, they may be structured in accordance with anysuitable protocol (e.g., UDP, TCP), and have any suitable payloadstructure (e.g., RTP/UDP, etc.). Furthermore, suitable encapsulation,tunneling or encryption may be provided. In addition, it is alsopossible to create additional (RTP) packets to be sent to the externalsystem 72. These additional (RTP) packets may carry missing orreconstructed control information (e.g., codec-identifying information)that may assist the external system 72 in successfully decoding thecoded video data.

In some embodiments, the output socket of payload redirection module 82may be configured to send the new packets to a console via an outputport (e.g., an Ethernet port or network interface card). Alternativelyor in addition, the new packets may be sent onto an external network(e.g., intranet or internet) via the output port. Alternatively or inaddition, the new packets may be saved to memory by a local recordingmodule. As such, the coded video data conveyed by the new packets can bedecoded by a device that ultimately views or processes the coded videodata further downstream or at a later time.

The above payload redirection process included various steps that checkto see if the current flow appears in the aforementioned table 84, whichis dynamically updated as new flows are discovered to be associated withcoded video data, and as old flows fail to produce coded video dataafter a certain timeout period. In other embodiments, this check can beperformed not against all flows in the table 84, but rather against apre-defined set of flows. In other words, a comparison is done against alimited set of flows that may have been pre-defined as being ofinterest. These pre-defined set of flows may be supplied by the externalsystem 72 or by a user via the data network, and the tests performed bythe codec detection module 78 may be performed only if the current flowappears in the pre-defined set of flows. This could further reduce thebandwidth of the signal traveling from the surveillance module 70 to theexternal system 72.

It is also within the scope of the present disclosure for the codecdetection module 78 to be configured to send control information to thepayload redirection module 82 so as to control the payload redirectionmodule's creation of new packets for a given flow.

Case 2—Modification (“Active”)

With reference now to FIG. 10, in this embodiment, the surveillancemodule 70 replaces coded video data 22 carried by certain packetstraveling from the switch 16 to the VMS 26 with replacement coded videodata. This allows the substitution of video images into in-progressvideo streams at line speed. The substituted video images may originateeither from a video data bank from a video editor (not shown), forexample. In clandestine applications, the introduction of replacementcoded video data potentially fool users of the VMS 26 into continuing tobelieve that the video streams displayed by the VMS 26 are thosereceived from the cameras 12 ₁ . . . 12 _(A). In privacy-preservingapplications, this embodiment has the potential to obfuscate features(e.g., human faces) of the video streams that are provided to the VMS 26before the VMS 26 has a chance to store or display these features. Inthe following description of this example embodiment, only packetstraveling from the cameras 12 ₁ . . . 12 _(A) to the VMS 26 arecaptured, but those skilled in the art will find it within their purviewto apply these teachings to the opposite direction of travel.

Those skilled in the art will appreciate that the packets (e.g., IPpackets) that arrive at the surveillance module 70 may arrive at a highspeed (packet bit rate), and in some cases may include coded video datafrom one or more of the cameras 12 ₁ . . . 12 _(A), whereas in somecases they may include data that is not coded video data (e.g., controlinformation related to the cameras or data not related in any way to thecameras). The role of the surveillance module 70 in this embodiment isto identify, among the unpredictable morass of received original IPpackets, those containing coded video data and to replace some of thecoded video data on-the-fly with replacement coded video data that hasbeen encoded using the same codec as the coded video data in theoriginally received packet. This is not a simple task, as thesurveillance module 70 does not know a priori whether a given receivedpacket is a video-data-containing packet, let alone the codec that mayhave been used.

To this end, the surveillance module 70 is configured for receiving viaan input port a signal from the switch 16. The received signal is fed toa packet capture module 76. The packet capture module 76 is configuredto detect packets (e.g., IP packets) in the received signal. The packetsdetected by the packet capture module 76 are fed to both (i) a codecdetection module 78 and (ii) a payload modification module.

The codec detection module 78 carries out a similar codec detectionprocess 222 as was previously described with reference to the flowchartin FIG. 5. As previously described, a purpose of the codec detectionprocess 222 may be to discriminate between two main possibilities: (1)the received packet contains coded video data and (2) the receivedpacket does not contain coded video data. If, on the one hand, thepacket contains coded video data, then this means that the packet wouldadhere to a certain payload structure, and therefore the outcome of thecodec detection process 222 will be the current flow as well as thecorresponding payload structure. In addition, this particular embodimentof the codec detection process 222 also outputs the detected codec typefor the current flow. If, on the other hand, the packet does not containcoded video data (or contains coded video data that the codec detectionmodule 78 is unable to detect), then the outcome of the codec detectionprocess 222 would be either nothing or an indication of the current flowtogether with an indication that there was no detectable coded video 22in the packet.

Accordingly, with reference to FIG. 5A, a packet is received at step1010A and, at step 1020A, the codec detection process 222 includesdetermining whether the originally received packet is an IP packet. Ifnot, the packet is ignored or discarded. If yes, the next step is step1030A, whereby the codec detection process 222 includes attempting tocheck the lower-layer communication protocol used by the IP packet. Ifnone can be identified, then the process may terminate. In case thecommunication lower-layer protocol is UDP, the codec detection process222 proceeds to step 1040A, whereby a codec autodetect sub-process 210is carried out for the current flow with the input variable “UDP”; incase the lower-layer communication protocol is TCP, the codec detectionprocess 222 proceeds to step 1050A, whereby the codec autodetectsub-process 210 is carried out for the current flow with the inputvariable “TCP”.

The codec autodetect sub-process 210 for UDP and TCP will be describedin greater detail later on. For now, those skilled in the art shouldappreciate that the codec autodetect sub-process 210 may include someinitial processing of the payload of the received packet to determinethe payload structure of the received packet (which is really only acandidate payload structure at this stage), followed by some processingof the payload of the received packet in accordance with the candidatepayload structure, which includes processing the payload of the receivedpacket in accordance with one or more tests, each test associated with aspecific codec. If a given test of the one or more tests is passed, anassociation is created between the current flow and at least thecandidate payload structure. The output of the codec autodetectsub-process 210 (carried out at step 1040A or step 1050A) is thusidentity of the candidate payload structure as well as the codec forwhich the test was ultimately passed (if any). In the presentembodiment, the codec detection process 222 then proceeds to step 1060A,whereby it creates an association between (i) the current flow and (ii)the candidate payload structure and (iii) the codec associated with thetest that was passed.

The aforementioned association between the current flow and thecandidate payload structure and the codec associated with the given testmay be sent directly to the payload modification module 198.Alternatively or in addition, a table 86 that stores this informationmay be populated by the codec detection module 78 and made accessible tothe payload modification module 198. Specifically, the table 86 mayinclude records each containing a “flow” field that specifies theparameters of a given flow (e.g., any suitable combination of sourceaddress, destination address, source port, destination port, MACaddress, etc.), as well as a “payload structure” field that specifiesthe candidate payload structure (e.g., RTP/UDP, RTSP-I/TCP, HTTP MJPEGmultipart) found to be associated with the given flow and a “codec”field that specifies the codec type (e.g., H263, MPEG4, H.264, H.264bitstream mode, H.265, AAC, PCM, MJPEG) associated with the given flow.In some cases, more than one codec may be associated with the givenflow. For example, in the case of RTSP-I, it may be necessary to provideplural “codec” sub-fields associated with plural RTP headers that mayoccupy the payload of the same TCP packet.

The information in the table 86 provides the payload modification module198 with an indication of those flows that contain coded video data thatmay be subject to replacement, versus those flows for which there isinsufficient information to do this, in which case the received packetmust be passed along the communication link 28 untouched.

Operation of the payload modification module 198 is now described. Thepayload modification module 198 may be configured to carry out a payloadmodification process 154, which is now described in greater detail withreference to the flowchart in FIG. 11. In particular, at step 1210, apacket is received (it is assumed to be an IP packet, otherwise it maybe released untouched towards the VMS 26 via the output port of thesurveillance module 70). At step 1220, the current flow is determined(based on at least the header of the received packet but also possiblybased on part of the payload, as it may contain the header of anencapsulated packet). A video data bank table is then consulted at step1230 to see if the current flow is a candidate for replacement.Specifically, the video data bank table may include a collection ofrecords each including a “flow” field and a “replacement video” field.For a given record, the contents of the flow field indicates a givenflow and the contents of the replacement video field indicates a pointerto a set of pre-coded video data streams in the video data bank. Thepre-coded video data streams all include the same video data, exceptencoded in accordance with various codecs.

If step 1230 reveals that the current flow is not a candidate forreplacement, the payload modification process 154 may terminate bysimply sending the received packet to the VMS 26 in unmodified form(step 1240). However, if step 1230 reveals that the current flow is acandidate for replacement, the payload modification process 154 mayproceed to step 1250, where the payload structure and the codec for thecurrent flow are determined. This may be done by consulting the table 86(step 1260). There are two possibilities, either there is an entry forthe current flow in the table 86 or there is not. If there is no entry,then this means that despite the current flow being a candidate forreplacement, the payload modification module does not have enoughinformation to proceed. This could lead to the same scenario as if thecurrent flow were not a candidate for replacement, namely, the receivedpacket may be simply forwarded to the VMS 26 untouched. On the otherhand, if consulting the table 86 reveals that there is indeed a payloadstructure and a codec associated with the current flow, the next step isstep 1270, which involves retrieving replacement coded video data fromthe video data bank (at the location pointed to by video data banktable, as determined earlier). Of course, care should be taken to makesure that the replacement coded video data is coded with the same codecas the one that was found to be associated with the current flow.

Then, at step 1280, the payload modification module process includesreplacing the coded video in the received packet with the replacementcoded video data. This result in a new packet having the same payloadstructure as the received packet but in which the coded video data hasbeen replaced with the replacement coded video data. Finally, at step1290, the payload modification module process outputs the new packettowards the VMS 26 via the output port of the surveillance module 70.

It should be appreciated that some of the aforementioned steps may beperformed in a different order to achieve substantially similar effects.It should also be noted that the packets being output to the VMS 26include the same flow and other header information as the received IPpackets; this may make it more difficult for the recipient to notice anytampering in the case that the coded video has been replaced.Alternatively or in addition, copies of some or all of the packetsoutput to the VMS 26 may also be sent onto an external network (e.g.,intranet or internet) via another output port. Alternatively or inaddition, copies of some or all of the packets output to the VMS 26 maybe saved to memory by a local recording module.

In some embodiments, the payload modification module may be configuredto validate that, when replacing the coded video data of a given packet,it has already replaced the coded video of the first packet of thecorresponding frame of that same video stream. That is to say, should anew flow be detected mid-way through an image frame, the remainder ofthe image frame should be allowed to continue unchanged, and replacementcoded video data should only be introduced once the next frame hasbegun. This validation may be important in order to begin replacingvideo data on a key frame (i.e., not on an interpolated frame), so as tolimit the occurrence of detectable visual artefacts at the recipient.

The above description considers that the payload modification moduleobtains replacement coded video data from the video data bank. The videodata bank may be operationally coupled to the packet modificationmodule. The video data bank may be located within the surveillancemodule 70 or it may be accessible remotely, e.g., over a public datanetwork such as the Internet or over a private data network. The videodata bank furnishes replacement video streams to the payloadmodification module. In some embodiments, the replacement video streamsinclude multiple versions of the same video data, encoded usingdifferent codecs so as to be readily available in the desired format; inother embodiments, a real-time transcoding module (not shown) may beinvoked on demand. As an alternative to replacing the payload withreplacement video streams from the video data bank, a video streamedition module (not shown) may be used to on-the-fly edit regions ofinterest (e.g., facial features, license plates, etc.) within imagescarried in the coded video of various packets.

In addition, control information may be provided by the codec detectionmodule 78 to the payload modification module 198 in the form of, e.g., acontrol signal. The control information may indicate one or more of, forexample:

-   -   specific flows that are/are not candidates for replacement;    -   the type of replacement operation (e.g., replaced from video        data bank or edited by the video stream edition module)    -   the pointers to the pre-coded video data streams in the video        data bank that are to be used in lieu of original coded video,        for a particular flow (i.e., the video data bank table);    -   the timing of replacement (e.g., when it is to start or end).        Codec Autodetect Sub-Process

The codec autodetect sub-process 210 includes discovering a codec usedfor encoding video data carried by at least part of the payload of thereceived packets associated with a given flow. In the case of somecodecs, they are discoverable directly from the payload of each receivedpacket, whereas in the case of other codecs, they are discoverable onlyonce a “state” has been built up over multiple received packets for thegiven flow. In order to handle the latter case, the codec detectionmodule 78 may store a “state of codec discovery” for each of one or moreflows. As such, the codec detection module 78 carries out the presentcodec autodetect sub-process 210 for each received packet in order todiscover, where possible, the codec used to encode the video datacarried by the received packet or to build up the state of codecdiscovery for the flow to which the received packet belongs.

Reference is now made to FIG. 8A, which pertains to step 1040 of FIG. 5and step 1040A of FIG. 5A, whereby the codec autodetect sub-process 210is carried out for the current flow with the input variable “UDP”. Atstep 810, a protocol compliance verification is performed to determineif the received packet might have an RTP-over-UDP (or RTP/UDP) payloadstructure. This can be determined by checking certain bits in the UDPheader to determine the RTP version used and the RTP payload type. Ifthe protocol compliance verification determines that the received packetmight have a UDP/RTP payload structure, the codec autodetect sub-process210 proceeds to step 820 (in order to start testing for codecs),otherwise, the codec autodetect sub-process 210 may proceed to try toestablish the presence of other payload structures or may simplyterminate by returning that no codec was found in the UDP packet.

At step 820, the first of several tests on the RTP payload is performed.Each of the tests is associated with a specific codec. Each of the testsis considered “passed” if it successfully determines that the codec typeassociated with that test was used (or likely used) to encode video datain the RTP payload. The tests may be performed in series (in any order)or in parallel, or the code may be optimized so that the processing doneat certain steps is shared among multiple tests. For example, the testperformed at step 820, if passed, will return the codec type “H263”; thetest performed at step 830, if passed, will return the codec type“MPEG4”; the test performed at step 840, if passed, will return thecodec type “H.264”; the test performed at step 850, if passed, willreturn the codec type “H.264 bitstream mode”; and the test performed atstep 860, if passed, will return the codec type “H.265”. Still othertests can be performed for other codecs.

With specific reference now to step 820, which corresponds to the H263test, it can be described with reference to FIG. 8B. Specifically, atstep 822, the H263 test checks if the presumed H.263 video content ofthe RTP payload is the first packet of the current video frame. At step824, the H263 test checks if the presumed H.263 video content RTPpayload satisfies a minimum size requirement. The minimum sizerequirement may represent the smallest size in bytes required in orderto conclude that the RTP payload might be encoded using H263. If bothconditions are fulfilled, the H263 test performs step 826, which isperformed on the first usable bits of the RTP payload. The first usablebits of the payload of the RTP packet are determined by ignoring acertain number of bits from the beginning of the RTP payload (i.e., theH263 test begins to read after the number of bits to ignore).Consequently, if step 826 reveals that the first usable bits match aparticular signature (called “Picture Start Code”, e.g., the 22following bits: “‘0000 0000 0000 0000 1000 00”), the H263 test will havesuccessfully passed (“conclude H263”), and the codec autodetectsub-process 210 may terminate; otherwise the codec autodetectsub-process 210 continues with step 830.

With specific reference to step 830, which corresponds to the MPEG4test, it can be described with reference to FIG. 8C. Specifically, atstep 832, the MPEG4 test checks if the presumed MPEG4 content of the RTPpayload is be the first packet of the current video frame. At step 834,the MPEG4 test checks if the presumed MPEG4 video content of the RTPpayload satisfies a minimum size requirement. The minimum sizerequirement represents the smallest size in bytes required in order toconclude that the RTP payload packet might be encoded using MPEG4. Ifboth conditions are fulfilled, the MPEG4 test performs step 836, whichis performed on the first usable bits of the RTP payload. The firstusable bits of the payload are determined by ignoring a certain numberof bits to directly from the beginning of the payload 36 (i.e., theMPEG4 test begins to read after the number of bits to ignore).Consequently, if step 836 reveals that the first usable bits match aparticular signature according to MPEG4 standards (e.g., {0x00, 0x00,0x01}), the MPEG4 test will have successfully passed, and the codecautodetect sub-process 210 may terminate; otherwise the codec autodetectsub-process 210 continues with step 840.

With specific reference to step 840, which corresponds to the H.264test, it can be described with reference to FIG. 8D. Specifically, theH.264 test searches for several (e.g., 3) partial codec-identifyingpieces of information (so-called Network Abstract Layer Units (NALUs))that would be contained in the H.264 header if such were present in theRTP payload. The three NALUs can be referred to as Instantaneous DecoderRefresh (IDR), Picture Parameter Set (PPS) and Sequence Parameter Set(SPS). The detecting order may vary from one RTP packet to another, asloss of packet order may occur during transit on the network. Moreover,these NALUs may themselves be fragmented and contained in severalnon-consecutive packets. A “state of codec discovery” is therefore builtup over several packets. The H.264 test updates the state of codecdiscovery until all three NALUs have been found and discovery iscomplete. The state of codec discovery may be suspended between packets.Practically speaking, the state of codec discovery may be represented bya NALU flag that is set for each NALU as it is received, and reset undervarious conditions.

Turning now to the simplified flowchart in FIG. 8D, a check is done atstep 849 to see if the received packet is the first packet in thecurrent video frame. If not, the next step is step 842, but if yes, thestate is reset (e.g., NALU flags are reset) at step 849A and thefollowing step is step 842. At step 842, it is checked whether thereceived packet (the presumed RTP payload) appears to include one ormore NALUs (or portions of NALUs), which can be determined based onspecific bit patterns. If not, then a check is done at step 843 to seewhether the count of successive “useless” packets (i.e., not containingNALU information) exceeds a certain threshold (e.g., greater than 3, 5,10, etc. useless packets). If not, this means that the state continuesto be built up; therefore, the state is suspended and a next packet isawaited. However, it is possible that the correct codec is not H264,therefore the testing continues with step 850. Returning to step 843, ifthe count of successive useless packets determined at step 843 didexceed the threshold, then the next step is step 849B the state is reset(e.g., the NALU flags are reset). Testing continues with step 850.

On the other hand, if step 842 revealed that the presumed RTP payload ofthe received packet does appear to include one or more NALUs (orportions of NALUs), then then next step is step 846 where theappropriate NALU flag is set. A check is then done at step 848 to see ifall NALU flags have been set. If so, then the H264 test will havesuccessfully passed (“conclude H264”), and the codec autodetectsub-process 210 may terminate. If not all NALU flags have been set, thestate continues to be built up. The state is therefore suspended, butonly if the time taken to build the state is still within a “keep-aliveperiod” (e.g., 1-10 seconds) (see step 848A). Otherwise, the state isreset (e.g., the NALU flags are reset; see step 849B). In each case, itis possible that the correct codec is not H264 and therefore testingcontinues at step 850.

With specific reference to step 850, which corresponds to the H.264bitstream mode test, it can be described with reference to FIG. 8E.Specifically, at step 852, the H.264 bitstream mode test checks if thepresumed H.264 bitstream mode video content of the RTP payload is thefirst packet of the current video frame. At step 854, the H.264bitstream mode test checks if the RTP payload satisfies a minimum sizerequirement. The minimum size requirement represents the smallest sizein bytes required in order to conclude that the RTP payload might beencoded using H.264 bitstream mode. If both conditions are fulfilled,the H.264 bitstream mode test performs step 856, which is performed onthe first usable bits of the RTP payload. The first usable bits of theRTP payload are determined by ignoring a certain number of bits from thebeginning of the payload (i.e., the H.264 bitstream mode test begins toread after the number of bits to ignore). Consequently, if step 856reveals that the first usable bits match a particular signatureaccording to H.264 bitstream standards (e.g., 0x00, 0x00, 0x00, 0x01),the H.264 bitstream mode test will have successfully passed, and thecodec autodetect sub-process 210 may terminate; otherwise the codecautodetect sub-process 210 continues with step 860.

It is noted that H.264 bitstream mode is detected based on a particularsignature contained within the first bits of the RTP payload, whichmakes detection of H.264 bitstream mode simpler than for H.264 as theparticular signature is instantly recognizable (as opposed to cumulatingNALUs and maintaining/updating a state).

With specific reference to step 860, which corresponds to the H.265test, it can be described with reference to FIG. 8F. Specifically, theH.265 test searches for several (e.g., 4) partial codec-identifyingpieces of information (so-called Network Abstract Layer Units (NALUs))that would be contained in the H.265 header if such were present in theRTP payload. The three NALUs can be referred to as Instantaneous DecoderRefresh (IDR), Picture Parameter Set (PPS), Sequence Parameter Set (SPS)and Video Parameter Set (VPS). The detecting order may vary from one RTPpacket to another, as loss of packet order may occur during transit onthe network. Moreover, these NALUs may themselves be fragmented andcontained in several non-consecutive packets. A “state of codecdiscovery” is therefore built up over several packets. The H.265 testupdates the state of codec discovery until all four NALUs have beenfound and discovery is complete. The state of codec discovery may besuspended between packets. Practically speaking, the state of codecdiscovery may be represented by a NALU flag that is set for each NALU asit is received, and reset under various conditions.

Turning now to the simplified flowchart in FIG. 8F, a check is done atstep 869 to see if the received packet is the first packet in thecurrent video frame. If not, the next step is step 862, but if yes, thestate is reset (e.g., NALU flags are reset) at step 869A and thefollowing step is step 862. At step 862, it is checked whether thereceived packet (the presumed RTP payload) appears to include one ormore NALUs (or portions of NALUs), which can be determined based onspecific bit patterns. If not, then a check is done at step 863 to seewhether the count of successive “useless” packets (i.e., not containingNALU information) exceeds a certain threshold (e.g., greater than 3, 5,10, etc. useless packets). If not, this means that the state continuesto be built up; therefore, the state is suspended and a next packet isawaited. However, it is possible that the correct codec is not H265,therefore no conclusion is reached. Returning to step 863, if the countof successive useless packets determined at step 863 did exceed thethreshold, then the next step is step 869B the state is reset (e.g., theNALU flags are reset).

On the other hand, if step 862 revealed that the presumed RTP payload ofthe received packet does appear to include one or more NALUs (orportions of NALUs), then then next step is step 866 where theappropriate NALU flag is set. A check is then done at step 868 to see ifall NALU flags have been set. If so, then the H265 test will havesuccessfully passed (“conclude H265”), and the codec autodetectsub-process 210 may terminate. If not all NALU flags have been set, thestate continues to be built up. The state is therefore suspended, butonly if the time taken to build the state is still within a “keep-aliveperiod” (e.g., 1-10 seconds) (see step 868A). Otherwise, the state isreset (e.g., the NALU flags are reset; see step 869B).

It should be appreciated that the maximum number of useless packetsreferred to above in FIGS. 8D and 8F can be a user-defined variableequal to a number of packets that will be tolerated while the state ofcodec discovery is being built and which do not include usefulcodec-identifying information. In other words, it is possible that whensearching for a total number (e.g., 3 or 4) NALUs in a given frame, 2are received, and the third (or fourth) does not arrive. Each additionalpacket that does not contain the missing NALU (e.g., third or fourth)can be considered “useless”. If the number of “useless” packets reachesa certain level, then the codec autodetect sub-process 210 abandons thesearch, and reinitiates it when the next video frame starts to bereceived. Indeed, the third (or fourth) NALU may have been lost, and anycomputational resources spent searching for it would be wasted. Ofcourse, every time the search is abandoned, this leads to delay, and ifone is too quick to abandon the search (e.g., if the third or fourthNALU is just one packet away when abandoning the search), then one incura significant additional delay before the codec can be successfullyidentified for that flow. Thus, the maximum number of useless packets isa tradeoff between computational resources and delay,

It should be appreciated that each of the aforementioned tests isdesigned so as to determine whether the payload contains pre-determinedcodec-identifying information regarding a particular codec that issufficient to identify the particular codec as having been used toencode video data in the payload. If so, the test returns the particularcodec as an identified codec for the current flow associated with thereceived packet.

It should also be appreciated that in some embodiments, the videostreams must comply with certain general requirements including thebitrate (e.g., 8000 bps) and number of packets per second (e.g., 10packets per second) in order for the outcome of any of the tests 820 to860 to be considered valid.

Reference is now made to FIG. 8G, which pertains to step 1050 of FIG. 5and step 1050A of FIG. 5A, whereby the codec autodetect sub-process 210is carried out for the current flow with the input variable “TCP”. As apreliminary comment, those skilled in the art will appreciate that incontrast to UDP, which is a connectionless protocol commonly used totransmit in real-time media packets such as video and audio, TCPprovides a reliable and ordered way to transmit any type of content(e.g. audio, video, image, text, and so on). If the received packet is aTCP packet, it may or may not contain video data in its payload. Assuch, the codec detection process 222 is tasked with searching for videodata inside the received TCP packet at different levels.

First, at steps 872 and 874, a verification is performed to determine ifthe received TCP packet might have an HTTP MJPEG multipart payloadstructure (which would also imply that the codec type for the currentflow is MJPEG). This can be determined by comparing, at step 872,certain initial bits in the TCP payload against the introductory portionknown to be associated with an HTTP MJPEG multipart payload structure(e.g., --*\r\nContent-Type: image/jpeg\r\nContent-Lengthnr\n\r\n). Ifthe answer is yes, this would suggest that the following bytes of theTCP payload may include blocks of coded video data in JPEG format. Tocheck this, the codec detection process 222 proceeds to step 874, wherethe following bytes of the TCP payload (or the first bytes of the TCPpayload of the next received packet for the current flow) are comparedagainst another signature associated with JPEG images (e.g., “0xFF 0xD80xFF”). If the comparison is positive, it can be hypothesized that thepayload structure for the current flow is HTTP MJPEG multipart (andtherefore that the codec type for the current flow is of course MJPEG),and the codec detection process 222 may terminate.

Otherwise, the codec detection process 222 proceeds to perform averification to determine if the received TCP packet might have anRTSP-I/TCP payload structure. Specifically, the codec detection process222 verifies, at step 880, if the presumed RTSP-I packet thought to becarried in the payload of the TCP packet starts with a particularsignature proper to the header of an RTSP-I packet (e.g., “0x24 0x00”).If yes, then at step 882, the codec detection process 222, presumingthat the packet carried in the payload of the TCP packet is an RTSP-Ipacket, checks if the header of such packet satisfies a maximum sizerequirement based on the typical structure of an RTSP-I header (e.g.,less than 2048 bytes). If both conditions are met, the codec detectionprocess 222 executes a “TCP stream sequencing method” for locking ontothe current flow (i.e., to retrieve the correct order dictated by a TCPtransmission) and extracting the corresponding RTP payload from theRTSP-I packet.

To this end, a network congestion level is verified at step 884 and,depending on whether the congestion is low or high (this can be comparedagainst a threshold), one of two variants of the TCP stream sequencingmethod is used to retrieve the RTP payload, starting at either step 886(for low congestion) or step 894 (for high congestion):

At step 886 (for a low congestion network, with low memory footprint),it is confirmed if the current TCP packet is the first one associatedwith the current flow or if it corresponds to the next expected packetsequence number. If one of these conditions is satisfied, the TCP packetis consumed and the corresponding RTP payload is extracted and testedfor codecs at step 890 (see steps 820 . . . 860 previously describedwith reference to FIGS. 8B to 8F). If neither of these conditions issatisfied, the TCP stream sequencing method drops the flow lock at step892 (also referred to as “stream lock”), and ends. It is expected thatsooner or later (i.e., within the keep-alive period), the TCP packetswill transit in sequential order, due to the low congestion of thenetwork.

At step 894 (for a higher congestion network, with higher memoryfootprint), the TCP stream sequencing method stores several TCP packets(e.g., up to 20 TCP packets per flow, but this number is flexible)inside a table 894A ordered by sequence number. On receiving packetswith a sequence number already present in table 894A (which is verifiedat step 896), the previous instance and subsequent packets are droppedat step 898 (as this indicates a high likelihood of TCP retransmission);otherwise, the packet is stored in the table at step 899A. Next, the TCPstream sequencing method validates at step 899C that each packetsequence number follows the previous packet sequence number stored inthe table for the current flow. If so, the TCP packet is consumed atstep 888 and the corresponding RTP payload is extracted and tested forcodecs (see steps 820 . . . 860 previously described with reference toFIGS. 8B to 8F). If the number of packets exceeds a pre-defined limit(e.g., up to 20 packets) for the current flow (step 899E), the TCPstream sequencing method drops the flow lock at step 899F, and ends.

In a variant, step 899C is preceded by step 899B, where it is checkedwhether the received packet includes the acknowledge flag (i.e., ACKflag) and, if so, step 899D involves the TCP stream sequencing methodextracting and returning the RTP payload by assembling the TCP packetsup to the acknowledged packet sequence number (step 899D).

For either congestion level, it will be appreciated that theaforementioned TCP stream sequencing method extracts and returns an RTPpayload. The returned RTP payload is then processed by applying a set of(for MPEG4, H263, H.264, H.264 bitstream mode and H.265, for example,although other tests may be done for PCM, AAC and other codec types). Assuch, depending on the test results, the codec detection process 222 mayreturn one of these codec types as being associated with the currentflow.

Those skilled in the art will appreciate that in the case of RTSP-I, theTCP stream sequencing method may find multiple RTP headers in thepayload of the TCP packet and therefore the codec detection methodextracts and returns multiple RTP payloads, one for each of the RTPheaders. The codec that might be associated with each RTP payload needsto be detected individually. Therefore, for a given RTP header of agiven flow, the associated RTP payload is processed by applying theaforementioned series of tests described with reference to FIGS. 8Bthrough 8F (for MPEG4, H263, H.264, H.264 bitstream mode and H.265, forexample). As such, the codec detection process 222 may return multiplecodec types, one for each of the various RTP headers for the currentflow.

Those skilled in the art will appreciate that in the case of H.264 andH.265, it may be difficult for the external system 72 to decode thecoded video data without the complete set of NALUs, even if the type ofcodec is known. As such, the state of codec discovery is reconstructedfrom NALUs. In the aforementioned examples of H.264 and H.265 detectionwithin the codec autodetect sub-process (FIGS. 8D and 8F), the NALUs arelocated in the RTP payload of successive RTP packets associated with thesame flow. However, it will be appreciated that in some cases, some orall of the NALUs may be found elsewhere than in the RTP payloads of RTPpackets. For example, some or all of the NALUs (in particular, SPS andPPS) may be sent as part of a negotiation that occurs using controlmessages sent in accordance with a different protocol (e.g., RTSP overTCP). Specifically, the control messages may include an RTSP DESCRIBEmessage which, upon receipt by a camera, may cause the camera tonegotiate transmission of a video stream. This negotiation may containcrucial control information (e.g., some of all of the NALUs, such asSPS, PPS and/or VPS, for example) that may be detected and used by thesurveillance module 70.

To this end, the codec detection module 78 runs an out-of-band codeccontrol information reconstruction process for detecting NALUs that aresent using the RTSP protocol (rather than in the RTP payloads of RTPpackets) between the VMS and a given camera. This process is run if itis determined that the lower-layer communications protocol of theincoming IP packet is TCP (e.g., as part of FIGS. 8G and 8H).

Specifically, with reference to FIGS. 19 and 20, the out-of-band codeccontrol information reconstruction process comprises monitoring the TCPpackets from the cameras (block 1910). At block 1920, the out-of-bandcodec control information reconstruction process detects a particularresponse message from a given camera, e.g., starting with “RTSP/1.0 200OK”. This response message may be a response to an RTSP DESCRIBE messagefrom the VMS (e.g., MacA:IpA:TcpPortA) to the given camera(MacB:IpB:TcpPortB), although it need not be known that the RTSPDESCRIBE message was actually sent. Based on the response message (e.g.,starting with “RTSP/1.0 200 OK”), the out-of-band codec controlinformation reconstruction process (at block 1930) obtains the codectype (e.g., H.264 or H.265). This can be done upon detection of aspecific set of characters (e.g., “rtpmap”). Also, the out-of-band codeccontrol information reconstruction process (at block 1940) obtains oneor more NALUs (e.g., parameter sets such as SPS, PPS and/or VPS). Atblock 1950, the out-of-band codec control information reconstructionprocess creates a temporary entry in a key table for the identifiercombination {MacA, IpA, TcpPortA, MacB, IpB, TcpPortB} and associatesthe discovered NALUs (e.g., SPS, PPS, VPS) therewith.

The MAC address and the IP address of the VMS and the camera stay thesame when sending RTP packets with video data, but the ports may differ.As such, the identifier combination {MacA, IpA, TcpPortA, MacB, IpB,TcpPortB} is not necessarily a flow associated with video data. Rather,further listening for a reply to an RTSP SETUP message is required, asthis will reveal the ports used for transmitting RTP packets with videodata, which allows establishing the flow. If this does not occur withina certain amount of time (e.g., 10 seconds), the temporary entry in theaforementioned key table may be cleaned.

Accordingly, the out-of-band codec control information reconstructionprocess listens to messages from the cameras and attempts to detect aparticular second response message (e.g., starting with “RTSP/1.0 200OK”) within a time limit (block 1960). This second response message maybe a response to a matching RTSP SETUP message from the VMS, but sincethe RTSP SETUP message is not being listened to, all messages from thecameras must be listened to in order to identify one that has thecorrect same identifier combination {MacA, IpA, TcpPortA, MacB, IpB,TcpPortB}. If this does occur, the out-of-band codec control informationreconstruction process may then determine, from this second responsemessage (see block 1970), the port details of the ensuing RTP video datatransmission (e.g a unicast UDP stream, between the MacA:IpA:49466 andMacB:IpB:50016, for example). The flow is now known, e.g., {MacA, IpA,49466, MacB, IpB, 50016}, in this example.

At block 1980, the out-of-band codec control information reconstructionprocess may update the key table with the flow (for example, {Udp, MacA,IpA, 49466, MacB, IpB, 50016}, where 49466 and 50016 are the UDP portsthat will be used for video data transmission). The appropriate entry inthe key table will have been pre-filled with the previously discoveredNALUs (e.g., PPS, SPS, VPS) for what is now the known flow.

At this stage, these NALUs, in association with the flow, can becommunicated to the external system 72. At block 1990, this can be doneby the out-of-band codec control information reconstruction processsending a control message to the payload redirection module. The controlmessage may specify the aforementioned flow and its associated NALUs. Inresponse, the payload redirection module can be configured to create oneor more extra RTP packets, and these extra RTP packets may contain, intheir RTP payload, the relevant NALUs for the flow. These extra RTPpackets may be inserted ahead of or in between other RTP packets for theflow, which changes the sequence number of subsequent packets. Anadvantage of inserting the NALUs in the RTP payload of extra RTP packetsis that the external system 72 will receive all the necessary NALUs viathe RTP payload of successive RTP packets associated with the same flow,which could allow prompt decoding of the video data.

Transmission of the RTSP DESCRIBE message from the VMS to a given cameramay be done controllably by the VMS at a desired moment (known as“cycling”). If operation of the VMS cannot be controlled, transmissionof the RTSP DESCRIBE message from the VMS to a given camera (or a set ofcameras) may be triggered by interrupting the communication link 28 inorder to connect the tap 74.

In a variant, and with reference to FIG. 22, the out-of-band codeccontrol information reconstruction process comprises monitoring the TCPpackets (block 2210). At block 2220, the out-of-band codec controlinformation reconstruction process detects a particular response messagefrom a given camera, e.g., starting with “RTSP/1.0 200 OK”. Thisresponse message may be a response to an RTSP DESCRIBE message from theVMS (e.g., MacA:IpA:TcpPortA) to the given camera (MacB:IpB:TcpPortB).Based on this response message, the out-of-band codec controlinformation reconstruction process (at block 2230) obtains the codectype (e.g., H.264 or H.265). This can be done upon detection of aspecific set of characters (e.g., “rtpmap”). Also, the out-of-band codeccontrol information reconstruction process (at block 2240) obtains oneor more NALUs (e.g., parameter sets such as SPS, PPS and/or VPS). Atblock 2245, the out-of-band codec control information reconstructionprocess identifies a control URI (e.g.,rtsp://10.122.218.192:554/axis-media/media.amp/stream=0?videocodec=h265)and creates a hash (e.g., URI_hash). At block 2250, the out-of-bandcodec control information reconstruction process creates a temporaryentry in a key table for the identifier combination {MacA, IpA,TcpPortA, MacB, IpB, TcpPortB, URI_hash} and associates the discoveredNALUs (e.g., SPS, PPS, VPS) therewith.

The MAC address and the IP address of the VMS and the camera stay thesame when sending RTP packets with video data, but the ports may differ.As such, the identifier combination {MacA, IpA, TcpPortA, MacB, IpB,TcpPortB} is not necessarily a flow associated with video data. Rather,further listening for a suitable reply to an RTSP SETUP message isrequired, as this will reveal the ports used for transmitting RTPpackets with video data, which allows establishing the flow.

Accordingly, the out-of-band codec control information reconstructionprocess listens to the messages from the VMS 26 and from the cameras 12and attempts to find, within a time limit, a particular second response(from the camera side) message having a valid start (e.g., starting with“RTSP/1.0 200 OK”) and that follows a RTSP SETUP message (from the VMSside) having a matching URI hash (e.g., URI_hash). To this end, the RTSPSETUP message having a matching URI hash is detected (block 2255) andthen a response message having the correct same identifier combination{MacA, IpA, TcpPortA, MacB, IpB, TcpPortB} is detected (bock 2260).Thereafter, the out-of-band codec control information reconstructionprocess may then determine, from this second response message (see block2270), the port details of the ensuing RTP video data transmission. Theflow is now known.

At block 2280, the out-of-band codec control information reconstructionprocess may update the key table with the flow. The appropriate entry inthe key table will have been pre-filled with the previously discoveredNALUs (e.g., PPS, SPS, VPS) for what is now the known flow.

At this stage, these NALUs, in association with the flow, can becommunicated to the external system 72. At block 2290, this can be doneby the out-of-band codec control information reconstruction processsending a control message to the payload redirection module. The controlmessage may specify the aforementioned flow and its associated NALUs. Inresponse, the payload redirection module can be configured to create oneor more extra RTP packets, and these extra RTP packets may contain, intheir RTP payload, the relevant NALUs for the flow. These extra RTPpackets may be inserted ahead of or in between other RTP packets for theflow, which changes the sequence number of subsequent packets. Anadvantage of inserting the NALUs in the RTP payload of extra RTP packetsis that the external system 72 will receive all the necessary NALUs viathe RTP payload of successive RTP packets associated with the same flow,which could allow prompt decoding of the video data.

The above description provides a specific non-limiting example of aprocess that may be generally described with reference to FIG. 21,whereby a data stream containing video data for a given flow is received(step 2110), a control stream that contains codec-identifyinginformation associated with the given flow is also received (step 2120,which could be before of after step 2110, or at the same time); and anoutput stream is created at step 2130. The output stream includespackets containing the video data from the data stream and additionalpackets containing the codec-identifying information.

In some embodiments, the process is not aware a priori that it hasreceived a control stream (such as an RTSP stream), let alone a controlstream containing codec-identifying information. As such, the processmay include an additional step of determining that a received packet ispart of a control stream (e.g., by looking for certain key informationin the header or payload of the packet and/or by looking for a certainpre-determined number of packets containing predetermined markers). Ingeneral, the codec detection module can be configured to attempt to (i)find information in an incoming stream of packets that would beindicative of the stream of packets being a control stream and/or (ii)reconstruct a portion of a control stream from the stream of packets,and determining that the stream of packets is the control stream if theattempting is successful.

Once the process has determined that a control stream has been received,the process can determine that the control stream containscodec-identifying information. In other words, a two-step process may beinvolved, where the second step (detecting codec-identifying informationin a given stream) is conditional upon success of the first step(confirming that the given stream is a control stream).

Also, the above disclosure has described an out-of-band codec controlinformation reconstruction process that obtains the codec type andcodec-identifying information in an out-of-band way (e.g., in packets ofa control stream such as RTSP packets) as well as a codec autodetectsub-process that determines a codec type by building a codec state basedon in-band information (e.g., from codec-identifying information inpackets of a data stream such as RTP packets). It should be appreciatedthat in some embodiments, both methods may be used together. Forexample, the codec detection module may attempt to detect the codec typefrom packets in an incoming control stream and, if it is unsuccessful(e.g., after a certain period of time has elapsed or after a certainnumber of packets has been received, processed or tested), then thecodec detection module may proceed to attempt to detect the codec typefrom packets in an incoming data stream for codec-identifyinginformation that would allow it to determine that a codec of a certaincodec type is being used. The opposite may also be done. The codecdetection module may also be configured to apply both methods inparallel so as to more quickly determine the codec type. Also, the codecdetection module may be configured to require that both methods becarried out and that both methods yield the same result before declaringthat a certain codec type has been successfully detected. If detectionof a certain type of video codec type is successful for a given flow,the codec detection module may be configured to associate the certaintype of video codec with the given flow.

As such, considering that the data stream identifies a source and adestination of the data stream and that the control stream alsoidentifies a source and a destination of the control stream (which couldbe the same or different as the source/destination of the data stream),the surveillance module that implements the codec detection module maybe neither the source nor the destination of the data stream or of thecontrol stream. Moreover, the surveillance module may be remote from thesource and destination of the data and control streams.

Those skilled in the art will appreciate that in some cases the codecdetection process 222 described above relies on heuristics and knowledgeregarding video stream structures and behavioral events, and thereforecertain of the above examples may be used as a training dataset toimprove reliability and precision through machine learning of aprobabilistic model. Such a model may generalize the outcome of thecodec detection process 222 by capturing a latent (i.e., non-observedproperties in data) and observable set of variables toward aclassification effort (i.e., assume codec based on similarities detectedbetween an input video stream and the current training dataset). Theprobabilistic model may be applied through a complementary routine tothe codec detection process 222. In addition to learning the variousvideo stream data structures from the training dataset, the model mayalso output a confidence level for the type of codec based on events(e.g., packet sequencing, TCP fragmentation, congestion, and so on)occurring on the network.

A potential advantage of this complementary routine may be the adaptiveaspect of an always-expanding training dataset. In other words, the morevideo streams are analyzed by the complementary routine, the more theprobabilistic model may be efficient and adapt to variances in the typesand data structures present in various video streams. The complementaryroutine may also prevent the codec discovery process fromfalse-negatives and false-positives through unsupervised learning (i.e.,unlabeled classification).

Generalized Approach

With reference to FIG. 23A, a surveillance module 2300 (which can alsobe surveillance module 70) is operable to access a plurality of networkpackets 2302 using passive or active monitoring techniques as describedherein. The packets carry 2302 data originating from various sources anddestined for various destinations (e.g., Alice and Bob). The packets2302, e.g., IP packets, may be connection-oriented (e.g., TCP) orconnectionless (e.g., UDP) or a combination thereof.

The data carried by the packets may be of various types, including text,images, audio, video (live ore pre-recorded), web browsing commands, IoT(Internet of Things) sensor data, other control data, and still othertypes of data and combinations thereof. Each of these types of data maybe formatted in accordance with a certain format. In the particular caseof packets carrying video, the video may be formatted into video frames.Various protocols spanning various layers of the OSI communication modelmay ensure proper transport of data by IP packets. One example that isused often in the transport of video data is RTP, which spans layers 4,5, 6 and 7 of the OSI model.

Certain ones of the packets 2302 can be considered as related to oneanother. One non-limiting example of relatedness is where packetsoriginate from the same source IP address and are destined for the samedestination IP address. Another example of packets being considered asrelated to one another is where they share more than just the sameorigin and destination IP addresses, but also the same port. Anotherexample of packets being related to one another is where a designatedportion of their respective headers is identical.

Other ways of defining or establishing relatedness are possible. Forexample, there may be a hierarchical nesting of “relatedness”:

-   -   At the Network layer (OSI layer 3): relatedness may be        established based on source and destination.    -   At the Transport layer (OSI layer 4): relatedness may be        established based on source and destination ports, as well as        the packet sequence ordering. Sequencing is particularly        relevant in RTP when using UDP transport.    -   At the Session layer (OSI layer 5): relatedness may be        established by the SSRC (synchronization source identifier),        which identifies which session the stream belongs to. For        example, in a VoIP telephone conference call, each participant        may have unique Network-layer and Transport-layer information        but would share the same Session-layer identifier (e.g., SSRC)        which identifies the conference call that the participant        belongs to and how to resynchronize the component streams of the        conference call.    -   The Presentation layer (OSI layer 6): relatedness may be        established based on information about how to go from a network        representation to an internal representation of various data        types, for example:        -   Whether integers Big Endian or Little Endian,        -   Whether strings is null terminated or Length-Value        -   Whether strings are ASCII, EBCEDIC, UTF, etc.    -   The Application layer (OSI layer 7): relatedness may be        established based on information about the encoding of the        media: MPEG4, MJPEG, etc.

Deeper in the media encoding one can find structures that may or may notbe “of interest”, such as video frames, or simply “frames”. A frame is acollection of various data structures needed to reconstruct one picturein a sequence of pictures. Structures of interest may include I-Frames,B-Frames and P-Frames, to name a few non-limiting possibilities. FIG.23B illustrates several frames as two groups of related frames 2350 and2360. Frames 2350 are related to one another and frames 2360 are relatedto one another, but frames 2350 may be unrelated to frames 2360. Frames2350, 2360 may be related to the packets 2302 as one frame per packet,multiple frames per packet or multiple packets per frame. Thesurveillance module 2300 may utilize various techniques, such as headerinspection, to analyze the packets 2302 and identify related frames.

In the lower portion of FIG. 23B, frames 2350 have been groupedtogether, and frames 2360 have been grouped together, for ease ofreference.

Considering now specifically the case of frames 2350 (frames 2360 couldhave been chosen just as easily), these frames may carry video data inaccordance with a variety of “structures”. A non-limiting example of a“structure” may be a protocol format, such as HTTP, H.264, H.263, H.265,MJPEG, SIP, etc. Another non-limiting example of a “structure” may be apayload structure for streaming video, such as “RTP/UDP” or“RTSP-I/TCP”.

Reference is now made to FIGS. 24A through 24F, which depict variousscenarios in which portions of certain frames are in accordance with(i.e., follow or abide by) a particular structure, regardless of thedata type. Accordingly:

-   -   FIG. 24A depicts a scenario in which a portion 2450 of the frame        2400 (in this case, the entirety of the frame 2400) is in        accordance with a first particular structure.    -   FIG. 24B depicts a scenario in which a portion 2452 of the frame        2402, that is less than the entirety of the frame 2402, is in        accordance with a second particular structure.    -   FIG. 24C depicts a scenario in which portions 2454A, 2454B of        each of two consecutive frames 2404 and 2406 are in accordance        with a third particular structure.    -   FIG. 24D depicts a scenario in which portions 2456A, 2456B 2456C        of each of several frames 2412, 2414, 2416 are in accordance        with a fourth particular structure.    -   FIG. 24E depicts a similar scenario to the one shown in FIG.        24D, except that there is an intermediate frame 2418 not        containing any portion that is in accordance with the structure,        such that the particular structure spans multiple frames that        are not necessarily contiguous.    -   FIG. 24F depicts a scenario in which different portions 2452,        2460 of a single frame 2422 are in accordance with respective        structures.

As such, a given one of the frames 2350 may include a portion(including, possibly, where the term “portion” means the entirety of theframe) that is in accordance with a given structure, or even multipleportions that are in accordance with multiple structures.

In accordance with embodiments of the present disclosure, thesurveillance module 2300 is adapted to carry out a method that involvesidentifying, in the set of related frames 2350 derived from packets2302, those portions of the related frames 2350 that are in accordancewith at least one “structure of interest”, and assembling such portionsinto at least one stream of new packets 2370 that are stored in memoryor sent elsewhere (such as an external entity, e.g., “Charlie”) forobservation/processing. This allows extraction and storage/delivery ofthe essence of what is considered “of interest” to the related frames2350.

In various cases, the at least one “structure of interest” may bespecified by a user, an external entity or selected by the surveillancemodule. The at least one structure of interest may comprise one or moreof the aforementioned structures, such as protocol formats and payloadstructures. In some embodiments, the at least one structure of interestmay comprise all structures of interest associated with a certain datatype, such as video data or web traffic, for example. In that case, thesurveillance module 2300 would be considered “on the lookout” for allvideo data or web traffic irrespective of which actual structure thevideo data or web traffic abides by.

The at least one structure of interest may be recorded in a memory andconceptualized as a table. FIG. 25 depicts a directed acyclic graph orpolytree 2500 showing a plurality of structures (and in some casessub-structures), each associated with a particular data type (in therow). The structure includes all branches leading down to a terminationof the polytree. An interest flag may encode a value indicative ofwhether the associated structure is a structure of interest. Forexample, when the interest flag is set to 1, this indicates that theassociated structure is a structure of interest and when the interestflag is reset to 0, this indicates that the associated structure is nota structure of interest. In terms of mapping to the polytree 2500, whenthe branches leading to a particular structure are all solid lines thiscorresponds to an interest flag that is set to 1 for the structure (orsub-structure) and when there is at least one dashed line along the paththis corresponds to an interest flag that is not set for that structure(or sub-structure). Thus, for example, for the video data type, thestructures that are structures of interest are H.263, H.264 and payloadsub-structure D2, but not payload sub-structure D1.

It is thus appreciated that the value of the interest flags collectivelydefine a set of structures of interest, and it is envisaged that thisset may change in size or composition over time as different structureseither become or cease to become structures of interest. In the polytreerepresentation of FIG. 25, branches may change from solid to dashed orvice versa. Also, new structures may be added to the graph 2500 (withoutnecessarily setting their interest flag), and obsolete structures may beremoved from the graph 2500.

Setting and resetting of the interest flags may be carried out by thesurveillance module 2300 under control of an external user (such asCharlie, for example).

Although not shown in FIG. 25, the tree 2500 may also representrelatedness field, which links together different structures of interestby requiring the same type of line (solid or dashed) for the suchstructures. The surveillance module 2300 may be configured to ensurethat the interest flags for related structures are always identical(either set or reset). This ensures that related structures of interestare kept together when required for the intelligibility of theredirected stream of new packets 2370.

As mentioned above, portions of the related frames 2350 that are inaccordance with at least one structure of interest are assembled into astream of new packets 2370 that are stored in memory or sent to anexternal entity for observation. Packets 2302 containing other portionsof the related frames 2350, as well as unrelated frames or no frames atall, may be redirected elsewhere or simply discarded by the surveillancemodule 2300.

Those skilled in the art will appreciate that although the structures ofinterest are known to the surveillance module 2300, it is not known apriori which of the received frames 2350 contain portions, if any, thatmight be in accordance with the structure(s) of interest.

It may therefore be desirable for the surveillance module 2300 toinspect and analyze the headers and payloads of the received packets2302 so as to determine whether the received packets 2302 containrelated frames 2350, and to determine whether the related frames 2350include portions that abide by any of the structures of interest. Thisinspection/analysis may include a process of identifying a structure inthe frames 2350 and determining whether it is a structure of interest,and/or a process of comparing certain bits, markers or statescorresponding to the structure of interest against bits, markers orstates found in the frames 2350.

Once a portion of a given frame 2350 is found to be in accordance with astructure of interest, such portion is extracted from the given frame2350 and assembled together with other successfully extracted portionsof other ones of the frames 2350 (or even of the same frame) into astream of new packets 2370, which can be recorded in memory (e.g., ondisk) or sent to Charlie (e.g., via a network interface). Any other datain the frames 2350 may be ignored or discarded by the surveillancemodule 2300. This can be analogized to the surveillance module 2300listening to a conversation between Alice and Bob, disregarding thesmall talk, and sending the essence of the conversation to Charlie.

In other cases, for example in the case where packets 2302 are relatedby a given flow, the surveillance module 2300 processes a particular oneof the packets to first determine or suggest a “candidate” payloadstructure of the particular packet (such as RTP/UDP or RTSP-1/TCP) andthen process at least part of the payload of the particular packet underthe assumption that the candidate payload structure is the true payloadstructure. This includes processing at least part of the payload of theparticular packet in accordance with one or more codec-specific testsand, if a given test of the one or more codec-specific tests is passed,this will confirm that the originally suggested candidate payloadstructure was correct, and an association may be created between theflow associated with the particular packet and the candidate payloadstructure. This association is recorded in memory and made available toa payload redirection module, which can learn that packets related tothis flow contain video and are structured a certain way (e.g.,RTP/UDP), thus allowing the video-carrying portions of the payload ofsuch packets to be extracted, assembled and transmitted to Charlie,without necessarily decoding the coded video in each packet. Thisprocess is described in further detail elsewhere in the presentdisclosure.

In some embodiments, the surveillance module 2300 may be under variousperformance or computing constraints. To this end, the surveillancemodule 2300 may be equipped with hardware and/or software configured tomonitor a particular computing condition such as bandwidth,memory/storage, temperature, processing power, etc. If the condition ismet, the surveillance module 2300 may take an action.

For example, if the condition that is met generally signals a worseningsituation (e.g., if available memory drops below a certain threshold, ifavailable bandwidth to the external entity drops below a certainthreshold, if network congestion rises above a certain threshold, iflatency rises above a certain threshold, etc.), the surveillance module2300 may be configured to reduce a quantity of data being assembled intothe stream of new packets 2370. In the opposite case, the surveillancemodule 2300 may be configured to increase the quantity of data beingassembled into the stream of new packets 2370.

One example of reducing the quantity of data being assembled into thestream of new packets 2370 is by way of reducing the number ofstructures that are considered structures of interest, i.e., byresetting certain interest flags in the polytree 2500. Conversely,certain interest flags (particularly those that had previously beenreset due to worsening conditions) may be set once conditions improveagain.

Consider now the situation where portions of the related frames 2350have been found to abide by a particular structure of interest and arebeing assembled into the stream of new packets 2370. Another way to paredown or “throttle” the quantity of data being assembled into the streamof new packets 2370 is to assemble only representative sub-portions ofsuch portions of the related frames 2350 into the stream of new packets2370.

By way of specific non-limiting example, in the case where the structureof interest is a certain video data format, and if it is determined thata bandwidth reduction is required, the surveillance module 2300 may beconfigured to throttle the quantity of video data being assembled intothe stream of new packets 2370. However, in accordance with certainembodiments of the disclosure, such throttling would not be donerandomly. Rather, throttling may be done in a deliberate anddeterministic manner, with a goal being to reduce visual artifacts seenby the external entity that receives the throttled stream forwarded bythe surveillance module 2300.

To this end, each structure of interest may be associated with its ownunique set of “instructions” when facing constraints of various kinds.Reference is now made to FIG. 26, which shows a table 2600 indicating,for each structure of interest, the representative portions to preserveunder different constraints. For example, in the case where thestructure of interest is structure “Structure A”, and when facingconstraint “Memory space reduction”, the surveillance module 2300 isconfigured to keep representative portions “XYZ”.

As a practical example, and with continued reference to table 2600 inFIG. 26, consider the structure of interest being an H.263 structure.When facing a constraint of “Bandwidth reduction”, the surveillancemodule 2300 is configured to preserve “every Xth video image” anddiscard the rest. This would result in using only approximately 1/X ofthe usual bandwidth, since only every Xth video image would find its wayinto the stream of new packets 2370 being sent to the external entity.

Another practical example involves a more complex codec type, such asH.264, whose structure comprises i-frames and p-frames. In this case,the representative portions are the i-frames. As such, when facing aconstraint of “Bandwidth reduction”, the surveillance module 2300 isconfigured to preserve “i-frames”, whereas p-frames may be ignored ordiscarded, e.g., until conditions improve. It is noted that the i-framescan be found at known locations within the frames 2350 without having todecode the packets themselves, thus the surveillance module 2300 cancontinue to operate at high speed, while significantly reducing thebandwidth and memory requirements of the stream of new packets 2370.

Those skilled in the art will appreciate that there may be severallayers of constraints and combinations of constraints, each associatedwith its own respective set of instructions.

Memory Management

As new packets are received by the surveillance module 70 (or 2300), theidentifiers that define various flows may be stored in a memorycontainer 164 which, as shown in FIG. 9A, may include (or be representedas including) an array of records 161. Each of the records 161 has aplurality of fields including a flow field (which has source IP 156,source port 158, destination IP 160 and destination port 162sub-fields), a payload structure field, a codec field and a timestampfield. In this embodiment, upon identifying a codec for a packet havinga particular flow corresponding to the contents of the flow field of agiven record, the codec detection module 78 is configured to store thecorresponding payload structure and detected codec in the payloadstructure and codec fields, respectively, of the given record and tostore the current time in the timestamp field of the given record.

Since a newly discovered flow may require additional memory to beallocated, the codec detection module 78 is configured to perform amemory management process for managing the memory container 164 so as toprevent excessive and unfillable memory allocation requests. To thisend, the memory management process may utilize the notion of akeep-alive period, wherein the keep-alive period is a period of timeallotted for codec detection following detection of a new flow. Theeffect of the keep-alive period is to release memory when a codec is notidentified for a particular flow after a certain amount of time (e.g.,1-10 seconds, but possibly longer or shorter).

To this end, the memory container 164 may be stored as a global variableand the memory management process may be described with reference to theflowchart in FIG. 9B. At step 1300, the memory management processobtains the current time and compares it to the value of the timestampfield of each record of the memory container 164. This may be donesequentially or in parallel. At step 1310, if the difference computed isnot greater than a threshold (e.g., 1, 5, 10, 15 seconds), i.e., if thecurrent time and the value of the timestamp field for a given record arewithin the threshold of one another, the memory management processterminates for the present packet. However, if comparing the currenttime and the value of the timestamp field reveals a difference greaterthan the threshold for a given record, the memory management processproceeds to step 1320, where the control module carries out adeallocation of memory. Specifically, the memory space formerly taken upby the given record is freed up (released or deallocated). This may havethe advantage of preventing unnecessary consumption of memory for apotentially huge number of flows that ultimately may not be used for along time. Of course, this function can be optimized further so thatmemory allocation and deallocation is done asynchronously and/or lessoften and/or using larger blocks of memory (e.g., greater than a singlerecord at a time). Also, the memory management process may pre-allocateadditional memory as needs grow.

ADDITIONAL REMARKS

In view of the foregoing, it will be appreciated that the presentdisclosure provides various methods. These methods may be summarized inthe flowcharts of FIGS. 13-18.

With reference to FIG. 13, a computer-implemented method implementedwithin the codec detection module 78 is described. At step 1300, aplurality of packets is received. Each of the packets comprises a headerand a payload. It is noted that for each particular packet among thepackets the following additional steps are executed. At step 1310, theheader of the particular packet is processed to determine a flowassociated with the particular packet. Once, the payload of theparticular packet is processed to determine a candidate payloadstructure of the particular packet at step 1320, the payload of theparticular packet is further processed in accordance with the candidatepayload structure at step 1330. The latter includes payload processingin accordance with one or more codec-specific tests. If the test issuccessful, an association between the flow associated with theparticular packet and the candidate payload structure is created at step1340.

With reference to FIG. 14, a computer-implemented method carried out aspart of the codec autodetect process with an “instantaneous” approach isexplained below. At step 1410, the packet, comprising a header and apayload, is received. Next, at step 1420, a portion of the packet isprocessed to identify a flow associated with the packet. At step 1430,if it is determined that the payload contains pre-determinedcodec-identifying information regarding a particular codec that issufficient to identify the particular codec as having been used toencode video data in the payload, the particular code is identified(step 1440) and returned as an identified codec for the flow at step1450.

With reference to FIG. 15, a method carried out as part of the codecautodetect sub-process 210 with a “cumulative” approach is explainedbelow. At step 1510, a packet comprising a header and a payload isreceived. Then, a portion of the packet is processed to identify a flowassociated with the packet in step 1520. If it is determined that thepayload contains partial codec-identifying information regarding aparticular codec at step 1530, the partial codec-identifying informationis added to previously determined partial codec-identifying informationregarding the particular codec and, as a result, obtain cumulativecodec-identifying information regarding the particular codec (step1540). Following this, the particular codec is identified as having beenused to encode video data in the payload in step 1550. To finish, atstep 1560, the particular codec is returned as an identified codec forthe flow in case the cumulative codec-identifying information regardingthe particular codec is sufficient to identify the particular codec ashaving been used to encode video data in the payload.

With reference to FIG. 16, a computer-implemented method is shown. Atstep 1610, packets, comprising a header and a payload, are received.Specifically, for each of the packets, the following steps areperformed. In the first place, the header of the particular packet isprocessed at step 1620 in order to determine a flow associated with theparticular packet. If a payload structure based on the flow, the payloadstructure associated with transport of coded video data in the payloadof the particular packet, is successfully identified (step 1630), thecoded video data contained in the payload of the particular packet isrepackaged into a new packet at step 1640. Then the packet is forwardedto an external system (step 1650) and/or stored in memory (step 1660).

With reference to FIG. 17, a computer-implemented method implementedwithin the payload modification module and is described below. Thecomputer-implemented method comprises intercepting packets sent from asource device and destined for a destination device at step 1710. Thenfor a particular packet among the packets the following steps arecarried out: At step 1720, the header of the particular packet isprocessed to determine a flow associated with the particular packet. Ifa payload structure and a codec are identified based on the flow duringstep 1730, the payload of the particular packet is replaced withreplacement coded video data that has been encoded with the codec atstep 1740. Finally, at step 1750, the packet with the replacement codedvideo data is then released towards the destination device instead ofthe particular packet.

With reference to FIG. 18, a processor-implemented method carried outduring a memory management process is presented below. The memorymanagement process starts with receiving a plurality of packets, eachbelonging to one of a plurality of flows at step 1810. Upon receipt ofeach of the packet, the memory management process proceeds to performthe following actions. At step 1820, a record associated with the flowto which the packet belongs is searched in the memory. In case no recordis found within the memory, a portion of the memory is allocated to arecord associated with the flow to which the packet belongs (step 1825).Next, codec identification is attempted by processing the payload of thepacket. At this point, if a particular codec is successfully identified(step 1830) then the information regarding the particular codec isstored (step 1840) in the record associated with the flow to which thepacket belongs. If unsuccessful and a certain period of time has elapsedsince the portion of the memory has been allocated (verification at step1850), the portion of the memory is freed up at step 1860.

Also, it will be appreciated that certain embodiments or parts of thesurveillance module 70 can be implemented as hardware, firmware,software, or a combination thereof. For example, with reference to FIG.12, the codec detection module 78 and/or the payload redirection module82 and/or the payload modification module 198 may be implemented as anapparatus (e.g., a computing device) comprising a microprocessor 190 anda computer-readable program storage unit. An application program may betangibly stored in the program storage unit, and may encode the variousmethods and functions referred to above. The application program in theprogram storage unit, as well as operating system code, may be read andexecuted by the microprocessor 190, thereby to carry out the variousmethods and functions encoded in the application program. Themicroprocessor 190 may include one or more central processing units(“CPUs”) and/or graphics processing units (“GPUs”). An input/output(I/O) interface 196 allows the microprocessor 190 to communicate withthe outside world, be it with the tap 74, the console, a data network194 or a display 192.

It should also be appreciated that while the above description has beenfocused on video codecs, those skilled in the art would find it withintheir purview to apply the teachings herein to any media that is codedand decoded, and is transported using packets, including but not limitedto video and/or audio that is encoded/decoded by various video/audiocodecs.

The examples and language recited herein are intended for pedagogicalpurposes to aid the reader in understanding the principles of thedisclosed embodiments and concepts, and are to be construed as beingwithout limitation to such specifically recited examples and language.Moreover, statements herein reciting principles, aspects, andembodiments, as well as specific examples thereof, are intended toencompass both structural and functional equivalents thereof.Additionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents developed in thefuture, i.e., any elements developed that perform the same function,regardless of structure.

It should be appreciated that certain adaptations and modifications ofthe described embodiments can be made. Therefore, the above discussedembodiments are to be considered illustrative and not restrictive. Also,it should be appreciated that additional elements that may be needed foroperation of certain embodiments of the present disclosure have not beendescribed or illustrated as they are assumed to be within the purview ofthe person of ordinary skill in the art. Moreover, any feature of anyembodiment discussed herein may be combined with any feature of anyother embodiment discussed herein in some examples of implementation.Moreover, certain embodiments of the present disclosure may be free of,may lack and/or may function without any element that is notspecifically disclosed herein.

What is claimed is:
 1. A computer-implemented method, comprising:receiving packets, each of the packets comprising a header and apayload; for a particular packet among the packets: processing at leastthe header of the particular packet to determine a flow associated withthe particular packet; attempting to determine a payload structure basedon the flow, the payload structure associated with transport of codedvideo data in the payload of the particular packet; if the attempting issuccessful, repackaging coded video data contained in the payload of theparticular packet into a new packet and forwarding the new packet to anexternal system or storing the new packet in memory; wherein the flowassociated with the particular packet is the same as the flow associatedwith other ones of the received packets and different from the flowassociated with still other ones of the received packets.
 2. The methoddefined in claim 1, wherein the repackaging is carried out withoutdecoding the coded video data.
 3. The method defined in claim 1, whereinto determine a payload structure based on the flow, the method comprisesconsulting a memory element that stores an association between flows andrespective payload structures.
 4. The method defined in claim 3, whereinthe memory element is populated by a codec detection module.
 5. Themethod defined in claim 3, wherein the memory element further stores anassociation between flows and codec types.
 6. The method defined inclaim 1, wherein the header of the particular packet is a first headerand wherein the payload of the particular packet is a first payloadcomprising a second header and a second payload, wherein said processingat least the header of the packet to determine a flow associated withthe particular packet comprises processing the first header and thesecond header to determine the flow associated with the particularpacket.
 7. The method defined in claim 6, wherein the flow ischaracterized by a source address, a destination address, a source port,a destination port and/or a MAC address.
 8. The method defined in claim6, wherein the first header includes an indication of the particularpacket being formatted in accordance with the UDP communication protocolor the TCP communication protocol.
 9. The method defined in claim 6,wherein the new packet is configured such that a header of the newpacket conveys at least part of the first or second headers of theparticular packet, and the payload of the new packet conveys at leastpart of the second payload of the particular packet.
 10. The methoddefined in claim 1, wherein the payload structure is one of RTP/UDP,RTSP-I/TCP and MJPEG/TCP.
 11. The method defined in claim 1, wherein thecoded video data in the payload of the particular packet is encoded inaccordance with the H263, MPEG4, H.264 bitstream mode, H.264, H.265 orMJPEG codec type.
 12. The method defined in claim 1, wherein theparticular packet is received from a first device and wherein the headerof the particular packet indicates that the particular packet isdestined for a second device, the new packet being released to a thirddevice that is not the second device.
 13. The method defined in claim12, wherein the external system is the third device.
 14. The methoddefined in claim 12, wherein the receiving is done passively withoutinterrupting the packets traveling from the first device towards thesecond device.
 15. The method defined in claim 12, wherein the payloadof the particular packet is a first payload comprising a second headerand a second payload, wherein the new packet has a payload conveying atleast part of the second payload of the particular packet.
 16. Themethod defined in claim 15, wherein the first header of the particularpacket identifies the second device and wherein the header of the newpacket identifies the third device and not the second device.
 17. Themethod defined in claim 12, wherein the payload of the particular packetis a first payload comprising a second header and a second payload,further comprising packaging an entirety of the second payload into thepayload of the new packet.
 18. The method defined in claim 1, whereinthe packets are received from a network and wherein external system iscommunicatively isolated from the network.
 19. The method defined inclaim 1, wherein the external system comprises a video managementsystem.
 20. The method defined in claim 1, wherein the external systemcomprises a display.
 21. The method defined in claim 1, wherein thereceived packets include video packets, each of the video packetscontaining video data in the respective payload and specifying a flow inthe respective header, wherein the new packets are organized into videostreams, each of the video streams associated with a corresponding flow,wherein the new packets associated with a particular flow contain videodata associated with the particular flow in the corresponding payload.22. The method defined in claim 1, further comprising attempting todetermine a particular codec associated with the flow and decoding thecoded video data with the particular codec.
 23. The method defined inclaim 22, further comprising displaying on a display the decoded videodata.
 24. The method defined in claim 1, the packets being received at apacket bit rate, wherein the processing, attempting and repackaging arecollectively carried out at least as fast as the packet bit rate. 25.The method defined in claim 1, wherein the processing, attempting andrepackaging are collectively carried out in less time than the timebetween receiving a first one of the received packets and receiving theimmediately subsequent one of the received packets.
 26. The methoddefined in claim 1, further comprising determining whether the flow isin a predetermined set of flows, wherein the attempting to determine apayload structure based on the flow is performed only if the flow isdetermined to be in the predetermined set of flows.
 27. The methoddefined in claim 1, further comprising determining whether the flow isin a predetermined set of flows and ignoring or discarding the receivedpacket if the flow is determined not to be in the predetermined set offlows.
 28. A computing device comprising: a computer-readable programstorage unit comprising an application program and an operating systemcode; and a processor being configured to read and execute theapplication program so as to carry out a method that comprises:receiving packets, each of the packets comprising a header and apayload; for a particular packet among the packets: processing at leastthe header of the particular packet to determine a flow associated withthe particular packet; attempting to determine a payload structure basedon the flow, the payload structure associated with transport of codedvideo data in the payload of the particular packet; if the attempting issuccessful, repackaging coded video data contained in the payload of theparticular packet into a new packet and forwarding the new packet to anexternal system or storing the new packet in memory; wherein the flowassociated with the particular packet is the same as the flow associatedwith other ones of the received packets and different from the flowassociated with still other ones of the received packets.
 29. Acomputer-readable medium storing computer-readable instructions which,when executed by a processor, cause the processor to carry out a methodthat comprises: receiving packets, each of the packets comprising aheader and a payload; for a particular packet among the packets:processing at least the header of the particular packet to determine aflow associated with the particular packet; attempting to determine apayload structure based on the flow, the payload structure associatedwith transport of coded video data in the payload of the particularpacket; if the attempting is successful, repackaging coded video datacontained in the payload of the particular packet into a new packet andforwarding the new packet to an external system or storing the newpacket in memory; wherein the flow associated with the particular packetis the same as the flow associated with other ones of the receivedpackets and different from the flow associated with still other ones ofthe received packets.
 30. The computer-readable medium defined in claim29, wherein the repackaging is carried out without decoding the codedvideo data.
 31. The computer-readable medium defined in claim 29,wherein to determine a payload structure based on the flow, the methodcomprises consulting a memory element that stores an association betweenflows and respective payload structures, wherein the memory elementfurther stores an association between flows and codec types.
 32. Thecomputer-readable medium defined in claim 29, wherein the header of theparticular packet is a first header and wherein the payload of theparticular packet is a first payload comprising a second header and asecond payload, wherein said processing at least the header of thepacket to determine a flow associated with the particular packetcomprises processing the first header and the second header to determinethe flow associated with the particular packet.
 33. Thecomputer-readable medium defined in claim 29, wherein the particularpacket is received from a first device and wherein the header of theparticular packet indicates that the particular packet is destined for asecond device, the new packet being released to a third device that isnot the second device.
 34. The computer-readable medium defined in claim29, wherein the method further comprises attempting to determine aparticular codec associated with the flow and decoding the coded videodata with the particular codec.
 35. The computer-readable medium definedin claim 29, the packets being received at a packet bit rate, whereinthe processing, attempting and repackaging are collectively carried outat least as fast as the packet bit rate.
 36. The computer-readablemedium defined in claim 29, wherein the processing, attempting andrepackaging are collectively carried out in less time than the timebetween receiving a first one of the received packets and receiving theimmediately subsequent one of the received packets.
 37. Thecomputer-readable medium defined in claim 29, wherein the method furthercomprises determining whether the flow is in a predetermined set offlows, wherein the attempting to determine a payload structure based onthe flow is performed only if the flow is determined to be in thepredetermined set of flows.
 38. A computer-implemented method,comprising: receiving packets, each of the packets comprising a headerand a payload; for a particular packet among the packets: processing atleast the header of the particular packet to determine a flow associatedwith the particular packet; attempting to determine a payload structurebased on the flow, the payload structure associated with transport ofcoded video data in the payload of the particular packet; if theattempting is successful, repackaging coded video data contained in thepayload of the particular packet into a new packet and forwarding thenew packet to an external system or storing the new packet in memory;wherein the packets are received at a packet bit rate; and wherein theprocessing, attempting and repackaging are collectively carried out atleast as fast as the packet bit rate or in less time than the timebetween receiving a first one of the received packets and receiving theimmediately subsequent one of the received packets.
 39. The methoddefined in claim 38, wherein the repackaging is carried out withoutdecoding the coded video data.
 40. The method defined in claim 38,wherein to determine a payload structure based on the flow, the methodfurther comprises consulting a memory element that stores an associationbetween flows and respective payload structures, wherein the memoryelement further stores an association between flows and codec types. 41.The method defined in claim 38, wherein the header of the particularpacket is a first header and wherein the payload of the particularpacket is a first payload comprising a second header and a secondpayload, wherein said processing at least the header of the packet todetermine a flow associated with the particular packet comprisesprocessing the first header and the second header to determine the flowassociated with the particular packet.
 42. The method defined in claim38, wherein the particular packet is received from a first device andwherein the header of the particular packet indicates that theparticular packet is destined for a second device, the new packet beingreleased to a third device that is not the second device.
 43. The methoddefined in claim 42, wherein the receiving is done passively withoutinterrupting the packets traveling from the first device towards thesecond device.
 44. The method defined in claim 38, further comprisingattempting to determine a particular codec associated with the flow anddecoding the coded video data with the particular codec.
 45. The methoddefined in claim 38, further comprising determining whether the flow isin a predetermined set of flows, wherein the attempting to determine apayload structure based on the flow is performed only if the flow isdetermined to be in the predetermined set of flows.
 46. The methoddefined in claim 38, further comprising determining whether the flow isin a predetermined set of flows and ignoring or discarding the receivedpacket if the flow is determined not to be in the predetermined set offlows.
 47. A computing device comprising: a computer-readable programstorage unit comprising an application program and an operating systemcode; and a processor being configured to read and execute theapplication program so as to carry out a method that comprises:receiving packets, each of the packets comprising a header and apayload; for a particular packet among the packets: processing at leastthe header of the particular packet to determine a flow associated withthe particular packet; attempting to determine a payload structure basedon the flow, the payload structure associated with transport of codedvideo data in the payload of the particular packet; if the attempting issuccessful, repackaging coded video data contained in the payload of theparticular packet into a new packet and forwarding the new packet to anexternal system or storing the new packet in memory; wherein the packetsare received at a packet bit rate; and wherein the processing,attempting and repackaging are collectively carried out at least as fastas the packet bit rate or in less time than the time between receiving afirst one of the received packets and receiving the immediatelysubsequent one of the received packets.
 48. A computer-readable mediumstoring computer-readable instructions which, when executed by aprocessor, cause the processor to carry out a method that comprises:receiving packets, each of the packets comprising a header and apayload; for a particular packet among the packets: processing at leastthe header of the particular packet to determine a flow associated withthe particular packet; attempting to determine a payload structure basedon the flow, the payload structure associated with transport of codedvideo data in the payload of the particular packet; if the attempting issuccessful, repackaging coded video data contained in the payload of theparticular packet into a new packet and forwarding the new packet to anexternal system or storing the new packet in memory; wherein the packetsare received at a packet bit rate; and wherein the processing,attempting and repackaging are collectively carried out at least as fastas the packet bit rate or in less time than the time between receiving afirst one of the received packets and receiving the immediatelysubsequent one of the received packets.
 49. The computer-readable mediumdefined in claim 48, wherein the repackaging is carried out withoutdecoding the coded video data.
 50. The computer-readable medium definedin claim 48, wherein to determine a payload structure based on the flow,the method comprises consulting a memory element that stores anassociation between flows and respective payload structures, wherein thememory element further stores an association between flows and codectypes.
 51. The computer-readable medium defined in claim 48, wherein theheader of the particular packet is a first header and wherein thepayload of the particular packet is a first payload comprising a secondheader and a second payload, wherein said processing at least the headerof the packet to determine a flow associated with the particular packetcomprises processing the first header and the second header to determinethe flow associated with the particular packet.
 52. Thecomputer-readable medium defined in claim 48, wherein the particularpacket is received from a first device and wherein the header of theparticular packet indicates that the particular packet is destined for asecond device, the new packet being released to a third device that isnot the second device.
 53. The computer-readable medium defined in claim52, wherein the receiving is done passively without interrupting thepackets traveling from the first device towards the second device. 54.The computer-readable medium defined in claim 48, wherein the methodfurther comprises attempting to determine a particular codec associatedwith the flow and decoding the coded video data with the particularcodec.
 55. The computer-readable medium defined in claim 48, wherein themethod further comprises determining whether the flow is in apredetermined set of flows, wherein the attempting to determine apayload structure based on the flow is performed only if the flow isdetermined to be in the predetermined set of flows.
 56. Thecomputer-readable medium defined in claim 48, wherein the method furthercomprises determining whether the flow is in a predetermined set offlows and ignoring or discarding the received packet if the flow isdetermined not to be in the predetermined set of flows.